Ir al contenido

Documat

From Rule-Based to LLMs: A Performance and Variability Analysis of Galician Machine Translation Models

  • Autores: Sofía García González, Germán Rigau Claramunt Árbol académico, José Ramón Pichel Campos
  • Localización: Procesamiento del lenguaje natural, ISSN 1135-5948, Nº. 75, 2025 (Ejemplar dedicado a: Procesamiento del Lenguaje Natural, Revista nº 75, septiembre de 2025), págs. 349-369
  • Idioma: inglés
  • Títulos paralelos:
    • De los Sistemas basados en Reglas a los Modelos LLM: un Análisis de Rendimiento y Variabilidad de los Modelos de Traducción Automática para el Gallego
  • Enlaces
  • Resumen
    • español

      Este trabajo evalúa la traducción automática (TA) para los pares Inglés–Gallego, Español–Gallego y Portugués–Gallego, con el objetivo de identificar los modelos más efectivos en un dominio generalista. La evaluación abarca factores como calidad, variabilidad del rendimiento y tamaño. Los resultados muestran que, para Español–Gallego, los sistemas basados en reglas y los modelos bilingües superan a los modelos multilingües y LLMs. Sin embargo, en pares de lenguas más distantes, los modelos multilingües ofrecen mejores resultados. Se destaca la necesidad de más investigación para Portugués–Gallego en TA.

    • English

      This paper evaluates machine translation (MT) for English–Galician, Spanish–Galician, and Portuguese–Galician pairs, with the aim of identifying the most effective models for these language pairs in the general domain. The evaluation encompasses a range of factors, including model quality, performance variance and size. The assessment involves the evaluation of different open-source systems. The results obtained identify that, for Spanish–Galician, both a Rule-Based System and a bilingual Neural Machine Translation model outperform larger multilingual models and LLMs. However, for more distant language pairs, multilingual models demonstrate superior performance. The study underscores the necessity for further research in Portuguese–Galician pair.

  • Referencias bibliográficas
    • Artetxe, M. and H. Schwenk. 2019. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. Transactions...
    • Bao, E., A. Pérez, and J. Parapar. 2024. Adapting Large Language Models for Underrepresented Languages. In VII Congreso XoveTIC: impulsando...
    • Barrault, L., O. Bojar, M. R. Costa-jussà, C. Federmann, M. Fishel, Y. Graham, B. Haddow, M. Huck, P. Koehn, S. Malmasi, C. Monz, M. Müller,...
    • Campos, J., P. Gamallo, I. Alegria, and M. Neves. 2020. A Methodology to Measure the Diachronic Language Distance between Three Languages...
    • Carrino, C. P., J. Armengol-Estapé, O. d. G. Bonet, A. Gutiérrez-Fandiño, A. Gonzalez-Agirre, M. Krallinger, and M. Villegas. 2021. Spanish...
    • Costa-jussà, M. R., J. Cross, O. Çelebi, M. Elbayad, K. Heafield, K. Heffernan, E. Kalbassi, J. Lam, D. Licht, J. Maillard, et al. 2022. No...
    • Costa-jussà, M. R., M. Zampieri, and S. Pal. 2018. A neural approach to language variety translation. In Proceedings of the Fifth Workshop...
    • de Dios-Flores, I., C. Magariños, A. I. Vladu, J. E. Ortega, J. R. Pichel, M. García, P. Gamallo, E. Fernández Rei, A. Bugarín-Diz, M. González...
    • de Gibert Bonet, O., K. Kharitonova, B. Calvo Figueras, J. Armengol-Estapé, and M. Melero. 2022. Quality versus quantity: Building Catalan-English...
    • DeepSeek-AI. 2025. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning.
    • Fan, A., S. Bhosale, H. Schwenk, Z. Ma, A. El- Kishky, S. Goyal, M. Baines, O. Celebi, G. Wenzek, V. Chaudhary, et al. 2021. Beyond english-centric...
    • Federmann, C., T. Kocmi, and Y. Xin. 2022. NTREX-128 – news test references for MT evaluation of 128 languages. In Proceedings of the First...
    • Forcada, M. L., M. Ginestí-Rosell, J. Nordfalk, J. O’Regan, S. Ortiz-Rojas, J. A. Pérez-Ortiz, F. Sánchez-Martínez, G. Ramírez-Sánchez, and...
    • Gamallo, P., P. Rodríguez, I. de Dios-Flores, S. Sotelo, S. Paniagua, D. Bardanca, J. R. Pichel, and M. Garcia. 2024. Open generative large...
    • García, S. 2024. Enhaced apertium system: Translation into low-resource languages of Spain Spanish–Asturian. In B. Haddow, T. Kocmi, P. Koehn,...
    • García-Ferrero, I., R. Agerri, and G. Rigau. 2022. Model and data transfer for cross-lingual sequence labelling in zero-resource settings....
    • García-Mateo, C. and M. Arza. 2012. O idioma galego na era dixital – The Galician Language in the Digital Age. META-NET White Paper Series:...
    • Gilabert, J. G., C. Escolano, A. S. Savall, F. D. L. Fornaciari, A. Mash, X. Liao, and M. Melero. 2024. Investigating the translation capabilities...
    • González, S. G. and G. R. Claramunt. 2024. Study of the State of the Art Galician Machine Translation: English-Galician and Spanish-Galician...
    • Gonzalez-Agirre, A., M. Pàmies, J. Llop, I. Baucells, S. D. Dalt, D. Tamayo, J. J. Saiz, F. Espuña, J. Prats, J. Aula-Blasco, M. Mina, I....
    • Goyal, N., C. Gao, V. Chaudhary, P.-J. Chen, G. Wenzek, D. Ju, S. Krishnan, M. Ranzato, F. Guzmán, and A. Fan. 2022. The Flores-101 evaluation...
    • Kocmi, T., E. Avramidis, R. Bawden, O. Bojar, A. Dvorkovich, C. Federmann, M. Fishel, M. Freitag, T. Gowda, R. Grundkiewicz, B. Haddow, M....
    • Kudugunta, S., I. Caswell, B. Zhang, X. Garcia, C. A. Choquette-Choo, K. Lee, D. Xin, A. Kusupati, R. Stella, A. Bapna, and O. Firat. 2023....
    • Kwon, W., Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica. 2023. Efficient memory management for large...
    • Lample, G., A. Conneau, L. Denoyer, and M. Ranzato. 2017. Unsupervised machine translation using monolingual corpora only. arXiv preprint...
    • Lee, S., J. Lee, H. Moon, C. Park, J. Seo, S. Eo, S. Koo, and H. Lim. 2023. A survey on evaluation metrics for machine translation. Mathematics,...
    • Lu, Y., W. Zhu, L. Li, Y. Qiao, and F. Yuan. 2024. Llamax: Scaling linguistic horizons of llm by enhancing translation capabilities beyond...
    • Martins, P. H., P. Fernandes, J. Alves, N. M. Guerreiro, R. Rei, D. M. Alves, J. Pombal, A. Farajian, M. Faysse, M. Klimaszewski, P. Colombo,...
    • Mohammadshahi, A., V. Nikoulina, A. Berard, C. Brun, J. Henderson, and L. Besacier. 2022. SMaLL-100: Introducing shallow multilingual machine...
    • OpenAI, J. Achiam, S. Adler, et al. 2024. GPT-4 Technical Report.
    • Outeirinho, D. B., P. G. Otero, I. de Dios- Flores, and J. R. P. Campos. 2024. Exploring the effects of vocabulary size in neural machine...
    • Papineni, K., S. Roukos, T. Ward, and W.-J. Zhu. 2002. BLEU: A Method for Automatic Evaluation of Machine Translation. In Proceedings of the...
    • Popović, M. 2015. chrF: character n-gram Fscore for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation,...
    • Post, M. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers,...
    • Rei, R., J. G. C. de Souza, D. Alves, C. Zerva, A. C. Farinha, T. Glushkova, A. Lavie, L. Coheur, and A. F. T. Martins. 2022. COMET-22: Unbabel-IST...
    • Sanches, J., R. Ribeiro, and L. Coheur. 2024. From brazilian portuguese to european portuguese.
    • Snover, M., B. Dorr, R. Schwartz, L. Micciulla, and J. Makhoul. 2006. A study of translation edit rate with targeted human annotation. In...
    • Sutskever, I., O. Vinyals, and Q. V. Le. 2014. Sequence to sequence learning with neural networks.
    • Sánchez, J. M. R. and C. G. Mateo. 2022. Deliverable D1.15 Report on the Galician Language. Project deliverable; EU project European Language...
    • Tang, Y., C. Tran, X. Li, P.-J. Chen, N. Goyal, V. Chaudhary, J. Gu, and A. Fan. 2020. Multilingual translation with extensible multilingual...
    • Tiedemann, J. 2020a. The tatoeba translation challenge – realistic data sets for low resource and multilingual MT. In Proceedings of the Fifth...
    • Tiedemann, J. 2020b. The Tatoeba Translation Challenge–Realistic Data Sets for Low Resource and Multilingual MT. arXiv preprint arXiv:2010.06354.
    • Tiedemann, J. and S. Thottingal. 2020. OPUS-MT–Building open translation services for the World. In Proceedings of the 22nd Annual Conference...
    • Touvron, H., T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin,...
    • Vaswani, A., N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. 2023. Attention is all you need.
    • Xiang, J., H. Li, Y. Liu, L. Liu, G. Huang, D. Lian, and S. Shi. 2022. Investigating data variance in evaluations of automatic machine translation...
    • Zoph, B., D. Yuret, J. May, and K. Knight. 2016. Transfer learning for low-resource neural machine translation. In Proceedings of the 2016...
Fundación Dialnet

Mi Documat

Opciones de artículo

Opciones de compartir

Opciones de entorno