Overview of DIPROMATS 2024: Detection, Characterization and Tracking of Propaganda in Messages from Diplomats and Authorities of World Powers

• Autores: Pablo Moral Martín, Jesús M. Fraile, Guillermo Marco Remón, Anselmo Peñas Padilla , Julio Gonzalo Arroyo
• Localización: Procesamiento del lenguaje natural, ISSN 1135-5948, Nº. 73, 2024, págs. 347-358
• Idioma: inglés
• Títulos paralelos:
  • Overview de DIPROMATS 2024: detección, caracterización y seguimiento de la propaganda en mensajes de diplomáticos y autoridades de potencias mundiales
• Enlaces
  • Texto completo
• Resumen
  • español

    Este trabajo presenta DIPROMATS 2024, una tarea compartida incluida en IberLEF. Esta segunda edición introduce una tipología refinada de técnicas y un conjunto de datos más equilibrado para la detección de propaganda, además de una nueva tarea para la detección de narrativas estratégicas. El dataset de la primera tarea incluye 12.012 tweets en inglés y 9.501 en español de autoridades de China, Rusia, Estados Unidos y la Unión Europea. Los participantes abordaron tres subtareas por idioma: clasificación binaria de tweets propagandísticos, su agrupación en tres categorías de propaganda y su categorización en siete técnicas. La segunda tarea consiste en una clasificación multi-clase y multi-etiqueta para identificar a cuáles de las narrativas predefinidas pertenecen los tweets, siguiendo descripciones y ejemplos en inglés y español (aprendizaje few-shot). En total fueron evaluadas 40 ejecuciones de nueve equipos diferentes.

  • English

    This paper summarizes the findings of DIPROMATS 2024, a challenge included at the Iberian Languages Evaluation Forum (IberLEF). This second edition introduces a refined typology of techniques and a more balanced dataset for propaganda detection, alongside a new task focused on identifying strategic narratives. The dataset for the first task includes 12,012 annotated tweets in English and 9,501 in Spanish, posted by authorities from China, Russia, the United States, and the European Union. Participants tackled three subtasks in each language: binary classification to detect propagandistic tweets, clustering tweets into three propaganda categories, and fine-grained categorization using seven techniques. The second task presents a multi-class, multi-label classification challenge where systems identify which predefined narratives (associated with each international actor) tweets belong to. This task is supported by narrative descriptions and example tweets in English and Spanish, using few-shot learning techniques. 40 runs from nine different teams were evaluated.

• Referencias bibliográficas
  • Alisetti, S. V. 2024. Paraphrase Generator with T5, June.
  • Amigo, E. and A. Delgado. 2022. Evaluating extreme hierarchical multi-label classification. In Proceedings of the 60th Annual Meeting of the...
  • Bolsover, G. and P. Howard. 2017. Computational Propaganda and Political Big Data: Moving Toward a More Critical Research Agenda. Big Data,...
  • Bolt, N. 2012. The violent image: insurgent propaganda and the new revolutionaries. Hurst & Company, London. OCLC: 1233055622.
  • Cañete, J., G. Chaperon, R. Fuentes, J.-H. Ho, H. Kang, and J. Pérez. 2020. Spanish pre-trained bert model and evaluation data. Pml4dc at...
  • Colley, T. 2020. Strategic narratives and war propaganda. In P. Baines, N. O’Shaughnessy, and N. Snow, editors, The SAGE handbook of propaganda....
  • Conneau, A., K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, and V. Stoyanov. 2020. Unsupervised...
  • Da San Martino, G., S. Yu, A. Barrón-Cedeño, R. Petrov, and P. Nakov. 2019. Fine-Grained Analysis of Propaganda in News Article. In Proceedings...
  • Devlin, J., M.-W. Chang, K. Lee, and K. Toutanova. 2018. BERT: Pretraining of Deep Bidirectional Transformers for Language Understanding....
  • García-Díaz, J. A., P. J. Vivancos-Vicente, ´A. Almela, and R. Valencia-García. 2022. UMUTextStats: A linguistic feature extraction tool for...
  • Jacob, B., S. Kligys, B. Chen, M. Zhu, M. Tang, A. Howard, H. Adam, and D. Kalenichenko. 2017. Quantization and Training of Neural Networks...
  • Jiang, A. Q., A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. de las Casas, E. B. Hanna, F. Bressand, G. Lengyel,...
  • Jowett, G. and V. O’Donnell. 2015. Propaganda & persuasion. SAGE, Thousand Oaks, Calif, sixth edition edition.
  • Liu, Y., M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov. 2019. RoBERTa: A Robustly Optimized...
  • Miskimmon, A., B. O’Loughlin, and L. Roselle. 2013. Strategic narratives: communication power and the new world order. Number 3 in Routledge...
  • Moral, P. 2023. Restoring reputation through digital diplomacy: the European Union’s strategic narratives on Twitter during the COVID-19 pandemic....
  • Moral, P. 2024. A tale of heroes and villains: Russia’s strategic narratives on twitter during the covid-19 pandemic. Journal of Information...
  • Moral, P. and G. Marco. 2023. Assembling stories tweet by tweet: strategic narratives from Chinese authorities on Twitter during the COVID-19...
  • Moral, P., G. Marco, J. Gonzalo, J. Carrillo-de Albornoz, and I. Gonzalo-Verdugo. 2023. Overview of DIPROMATS 2023: automatic detection and...
  • Nguyen, D. Q., T. Vu, and A. Tuan Nguyen. 2020. BERTweet: A pre-trained language model for English Tweets. In Proceedings of the 2020 Conference...
  • Pérez, J. M., D. A. Furman, L. Alonso Alemany, and F. M. Luque. 2022. RoBERTuito: a pre-trained language model for social media text in Spanish....
  • Richards, J. 2023. The Use of Discourse Analysis in Propaganda Detection and Understanding. In Routledge Handbook of Disinformation and National...
  • Riessman, C. K. 2008. Narrative methods for the human sciences. Sage Publications, Los Angeles.
  • Sparkes-Vian, C. 2019. Digital Propaganda: The Tyranny of Ignorance. Critical Sociology, 45(3):393–409, May.
  • Zhang, X., Y. Malkov, O. Florez, S. Park, B. McWilliams, J. Han, and A. El-Kishky. 2023. TwHIN-BERT: A Socially-Enriched Pre-trained Language...