Overview of DETESTS-Dis at IberLEF 2024: DETEction and classification of racial STereotypes in Spanish - Learning with Disagreement

• Autores: Wolfgang S. Schmeisser Nieto, Pablo Castells Azpilicueta , Simona Frenda, Alejandro Ariza Casabona, Mireia Farrús i Cabeceran , Paolo Rosso , Mariona Taulé Delor
• Localización: Procesamiento del lenguaje natural, ISSN 1135-5948, Nº. 73, 2024, págs. 323-333
• Idioma: inglés
• Títulos paralelos:
  • Resumen de la tarea de DETESTS-Dis en IberLEF 2024: DETEcción y clasificación de eSTereotipos raciales en eSpañol - Aprendizaje con desacuerdos
• Enlaces
  • Texto completo
• Resumen
  • español

    Este artículo presenta un resumen de la tarea DETESTS como parte del workshop IberLEF 2024, dentro del congreso SEPLN 2024. Propusimos dos subtareas jerárquicas: En la subtarea 1, los participantes tuvieron que determinar la presencia de estereotipos raciales en oraciones. En la subtarea 2, de las oraciones etiquetadas con estereotipo, los participantes tuvieron que decidir si este era implícito. El dataset DETESTS-Dis contiene 12.111 oraciones de comentarios y tuits respondiendo a artículos de periódicos y bulos raciales sobre inmigración en español. 15 equipos se registraron para participar, de los cuales 6 enviaron predicciones de sistemas y 3 de ellos enviaron artículos. En este artículo presentamos información sobre los datasets de entrenamiento y de prueba, los sistemas utilizados por los participantes, las métricas de evaluación y sus resultados.

  • English

    This paper presents an overview of the DETESTS-Dis shared task as part of the IberLEF 2024 Workshop on Iberian Languages Evaluation Forum, within the framework of the SEPLN 2024 conference. We proposed two hierarchical subtasks: In subtask 1, participants had to determine the presence of stereotypes in the texts. For subtask 2, participants had to decide which texts labeled with stereotypes were implicit stereotypes. The DETESTS-Dis dataset contains 12,111 comment sentences and tweets in response to newspaper articles and verified racial hoaxes involving immigration in Spanish. 15 teams signed up to participate, 6 of which sent runs, and 3 of them sent their working notes. In this paper, we provide information about the training and test datasets, the systems used by the participants, the evaluation metrics of the systems and their results.

• Referencias bibliográficas
  • Allport, G. W., K. Clark, and T. Pettigrew. 1954. The nature of prejudice. Addison-wesley Reading, MA.
  • Amigó, E., J. Carrillo-de Albornoz, M. Almagro-Cádiz, J. Gonzalo, J. Rodríguez-Vidal, and F. Verdejo. 2017. Evall: Open access evaluation...
  • Amigo, E. and A. Delgado. 2022. Evaluating extreme hierarchical multi-label classification. In Proceedings of the 60th Annual Meeting of the...
  • Ariza-Casabona, A., W. S. Schmeisser-Nieto, M. Nofre, M. Taulé, E. Amigó, B. Chulvi, and P. Rosso. 2022. Overview of DETESTS at IberLEF 2022:...
  • Barbieri, F., J. Camacho-Collados, L. Espinosa Anke, and L. Neves. 2020. Tweet-Eval: Unified benchmark and comparative evaluation for tweet...
  • Bourgeade, T., A. T. Cignarella, S. Frenda, M. Laurent, W. S. Schmeisser-Nieto, F. Benamara, C. Bosco, V. Moriceau, V. Patti, and M. Taulé....
  • Cabitza, F., A. Campagner, and V. Basile. 2023. Toward a perspectivist turn in ground truthing for predictive computing. In Proceedings of...
  • Carrejón-Naranjo, M., M. Guerrero-García, J. Mata-Vázquez, and V. Pachón-Álvarez. 2024. I2C-Huelva at IberLEF-2024 DETESTS: Learning from...
  • Cañete, J., G. Chaperon, R. Fuentes, J.-H. Ho, H. Kang, and J. Pérez. 2020. Spanish pre-trained bert model and evaluation data. In PML4DC...
  • Chiril, P., F. Benamara, and V. Moriceau. 2021. “be nice to your wife! the restaurants are closed”: Can gender stereotype detection improve...
  • Chiruzzo, L., S. M. Jiménez-Zafra, and F. Rangel. 2024. Overview of IberLEF 2024: Natural Language Processing Challenges for Spanish and other...
  • Conneau, A., K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, É. Grave, M. Ott, L. Zettlemoyer, and V. Stoyanov. 2020. Unsupervised...
  • Cryan, J., S. Tang, X. Zhang, M. Metzger, H. Zheng, and B. Y. Zhao. 2020. Detecting gender stereotypes: Lexicon vs. supervised learning methods....
  • Fandiño, A. G., J. A. Estapé, M. Pàmies, J. L. Palao, J. S. Ocampo, C. P. Carrino, C. A. Oller, C. R. Penagos, A. G. Agirre, and M. Villegas....
  • Fersini, E., D. Nozza, and P. Rosso. 2018. Overview of the evalita 2018 task on automatic misogyny identification (ami). EVALITA Evaluation...
  • Fiske, S. 1998. Stereotyping, prejudice, and discrimination. In D. T. Gilbert, S. T. Fiske, & G. Lindzey (Eds.). The handbook of social...
  • Fokkens, A., N. Ruigrok, C. Beukeboom, G. Sarah, and W. Van Atteveldt. 2018. Studying muslim stereotyping through microportrait extraction....
  • González, A., D. S. García-Chicangana, and S. R. Galvis. 2024. UC3M-sas at Iber-LEF2024 DETESTS-Dis tasks. In Iber-LEF@ SEPLN.
  • Leonardelli, E., G. Abercrombie, D. Almanea, V. Basile, T. Fornaciari, B. Plank, V. Rieser, A. Uma, and M. Poesio. 2023. SemEval-2023 task...
  • Plaza, L., J. Carrillo-de Albornoz, R. Morante, E. Amigó, J. Gonzalo, D. Spina, and P. Rosso. 2023. Overview of exist 2023–learning with disagreement...
  • Rodríguez-Sánchez, F., J. C. de Albornoz, L. Plaza, J. Gonzalo, P. Rosso, M. Comet, and T. Donoso. 2021. Overview of EXIST 2021: sexism identification...
  • Rodríguez-Sánchez, F., J. C. de Albornoz, L. Plaza, A. Mendieta-Aragón, G. Marco-Remón, M. Makeienko, M. Plaza, J. Gonzalo, D. Spina, and...
  • Sánchez-Junquera, J. J., B. Chulvi, P. Rosso, and S. P. Ponzetto. 2021. How do you speak about immigrants? taxonomy and stereoimmigrants dataset...
  • Sanguinetti, M., G. Comandini, E. di Nuovo, S. Frenda, M. Stranisci, C. Bosco, T. Caselli, V. Patti, and I. Russo. 2020. Haspeede 2 @...
  • Sap, M., S. Gabriel, L. Qin, D. Jurafsky, N. A. Smith, and Y. Choi. 2020. Social bias frames: Reasoning about social and power implications...
  • Schmeisser-Nieto, W. S., M. Nofre, and M. Taulé. 2022. Criteria for the annotation of implicit stereotypes. In Proceedings of the Thirteenth...
  • Schmeisser-Nieto, W. S., P. Pastells, S. Frenda, and M. Taule. 2024. Human vs. machine perceptions on immigration stereotypes. In N. Calzolari,...
  • Tai, L. D. 2024. TaiDepZai999_UIT_AIC at IberLEF 2024 DETEST-Dis task: Classification of racial stereotypes in Spanish With Ensemble Learning...
  • Taulé, M., M. Nofre, V. Bargiela, and X. Bonet. 2024. Newscom-tox: a corpus of comments on news articles annotated for toxicity in spanish....
  • Uma, A., T. Fornaciari, A. Dumitrache, T. Miller, J. Chamberlain, B. Plank, E. Simpson, and M. Poesio. 2021. SemEval-2021 task 12: Learning...
  • Uma, A., T. Fornaciari, D. Hovy, S. Paun, B. Plank, and M. Poesio. 2020. A case for soft loss functions. Proceedings of the AAAI Conference...