NECOS: An annotated corpus to identify constructive news comments in Spanish

• Autores: Pilar López Úbeda, Flor Miriam Plaza del Arco , Manuel Carlos Díaz Galiano , María Teresa Martín Valdivia
• Localización: Procesamiento del lenguaje natural, ISSN 1135-5948, Nº. 66, 2021, págs. 41-51
• Idioma: inglés
• Títulos paralelos:
  • NECOS: Un corpus anotado para identificar comentarios constructivos de noticias en español
• Enlaces
  • Texto completo
• Dialnet Métricas: 1 Cita
• Resumen
  • español

    En este artículo presentamos un corpus de noticias y comentarios en español (NECOS). Estas noticias están publicadas en el periódico El Mundo en un período comprendido entre el 3 de abril y el 30 de abril de 2018. El corpus contiene un total de 10 noticias y 1.419 comentarios. Siguiendo un esquema de anotación, tres anotadores etiquetaron manualmente los comentarios como constructivos y no constructivos obteniendo un promedio de 78,97 usando el coeficiente de kappa de Cohen. En este estudio nos centramos en estudiar la constructividad y hacer la evaluación del corpus NECOS. Para abordar este objetivo, proponemos la experimentación con diferentes sistemas basados en Procesamiento del Lenguaje Natural usando aprendizaje automático: un clasificador tradicional y métodos recientes basados en Transformers. Concretamente, comparamos modelos multilingües con un modelo monolingüe entrenado para el español. Con ello, pretendemos demostrar la importancia de crear recursos entrenados para un idioma en particular. El modelo monolingüe evaluado en NECOS obtiene el mejor resultado alcanzando un 77,24% de macro-average F1.

  • English

    In this paper, we present the NEws and COmments in Spanish (NECOS) corpus, a collection of Spanish comments posted in response to newspaper articles. Following a robust annotation scheme, three annotators labeled the comments as constructive and non-constructive. The articles were published in the newspaper El Mundo between April 3rd and April 30th, 2018. The corpus is composed of a total of 10 news articles and 1,419 comments. Three annotators manually labeled NECOS with an average Cohen's kappa of 78.97. Our current focus is the study of constructiveness and the evaluation of the Spanish NECOS corpus. In order to address this goal, we propose a benchmark testing different machine learning systems based on Natural Language Processing: a traditional system and the novel Transformer-based models. Specifically, we compare multilingual models with a monolingual model trained on Spanish in order to highlight the need to create resources trained on a specific language. The monolingual model fine-tuning on NECOS obtain the best result by achieving a macro-average F1 score of 77.24%.

• Referencias bibliográficas
  • Aulamo, M. and J. Tiedemann. 2019. The OPUS resource repository: An open package for creating parallel corpora and machine translation services....
  • Pilar López-Úbeda, Flor Miriam Plaza-del-Arco, Manuel Carlos Díaz-Galiano, M.Teresa Martín-Valdivia 2019. Text classification using svm enhanced...
  • Cohen, J. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37–46.
  • Conneau, A., K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, and V. Stoyanov. 2019. Unsupervised...
  • Devlin, J., M.-W. Chang, K. Lee, and K. Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding.
  • Etim, B. 2017. The times sharply increases articles open for comments, using google’s technology. The New York Times, 13.
  • Fujita, S., H. Kobayashi, and M. Okumura. 2019. Dataset creation for ranking constructive news comments. In Proceedings of the 57th Annual...
  • Instituto Cervantes. 2018. El español: una lengua viva. https://cvc.cervantes. es/lengua/espanol_lengua_viva/pdf/ espanol_lengua_viva_2018.pdf.
  • Kolhatkar, V. and M. Taboada. 2017a. Constructive language in news comments. In Proceedings of the First Workshop on Abusive Language Online,...
  • Kolhatkar, V. and M. Taboada. 2017b. Using new york times picks to identify constructive comments. In Proceedings of the 2017 EMNLP Workshop:...
  • Kolhatkar, V., N. Thain, J. Sorensen, L. Dixon, and M. Taboada. 2020. Classifying constructive comments. arXiv preprint arXiv:2004.05476.
  • Kolhatkar, V., H. Wu, L. Cavasso, E. Francis, K. Shukla, and M. Taboada. 2019. The sfu opinion and comments corpus: A corpus for the analysis...
  • Lample, G. and A. Conneau. 2019. Crosslingual language model pretraining. arXiv preprint arXiv:1901.07291.
  • Liu, Y., M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov. 2019. Roberta: A robustly optimized...
  • McHugh, M. L. 2012. Interrater reliability: the kappa statistic. Biochemia medica: Biochemia medica, 22(3):276–282.
  • Napoles, C., A. Pappu, and J. Tetreault. 2017. Automatically identifying good conversations online (yes, they do exist!). In Eleventh International...
  • Napoles, C., J. Tetreault, A. Pappu, E. Rosato, and B. Provenzale. 2017. Finding good conversations online: The yahoo news annotated comments...
  • Pedregosa, F., G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas,...
  • Puri, S. and S. P. Singh. 2019. An efficient hindi text classification model using svm. In Computing and Network Sustainability. Springer,...
  • Swanson, R., B. Ecker, and M. Walker. 2015. Argument mining: Extracting arguments from online dialogue. In Proceedings of the 16th annual...
  • Vaswani, A., N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. 2017. Attention is all you need. In...