Transferencia de Tareas basada en Implicación Textual para la Clasificación de Textos en Catalán en Escenarios de Pocos Datos

• Autores: Marta Villegas Montserrat , Oier López de Lacalle Lecuona , Irene Baucells de la Peña, Blanca Calvo Figueras
• Localización: Procesamiento del lenguaje natural, ISSN 1135-5948, Nº. 71, 2023, págs. 165-177
• Idioma: español
• Títulos paralelos:
  • Entailment-based Task Transfer for Catalan Text Classification in Small Data Regimes
• Enlaces
  • Texto completo
• Resumen
  • English

    This study investigates the application of a state-of-the-art zero-shot and few-shot natural language processing (NLP) technique for text classification tasks in Catalan, a moderately under-resourced language. The approach involves reformulating the downstream task as textual entailment, which is then solved by an entailment model. However, unlike English, where entailment models can be trained on huge Natural Language Inference (NLI) datasets, the lack of such large resources in Catalan poses a challenge. In this context, we comparatively explore training on monolingual and (larger) multilingual resources, and identify the strengths and weaknesses of monolingual and multilingual individual components of entailment models: pre-trained language model and NLI training dataset. Furthermore, we propose and implement a simple task transfer strategy using open Wikipedia resources that demonstrates significant performance improvements, providing a practical and effective alternative for languages with limited or no NLI datasets.

  • Multiple

    El presente trabajo investiga una reciente técnica de aprendizaje zero-shot y few-shot, en que la tarea objetivo se reformula como un problema de implicación textual y se resuelve mediante un modelo de implicación (un modelo de lenguaje entrenado con un corpus de implicación o NLI (Natural Language Inference)), para abordar tareas de clasificación textual en catalán, una lengua con recursos limitados que dispone de un corpus de NLI de tamaño moderado. Comparamos su aplicación con los recursos en esta lengua frente a los multilingües, de tamaño muy superior. Así mismo, identificamos las ventajas y limitaciones de ambas aproximaciones y el impacto del tamaño y la lengua del modelo de lenguaje y corpus de NLI. Finalmente, implementamos una estrategia de transferencia de aprendizaje, empleando datos extraídos de Wikipedia, que consigue mejoras significativas y demuestra ser una opción interesante para lenguas que disponen de un corpus de NLI reducido o carecen de él.

• Referencias bibliográficas
  • Agerri, R., I. S. Vicente, J. A. Campos, A. Barrena, X. Saralegi, A. Soroa, and E. Agirre. 2020. Give your text representation models some...
  • Armengol-Estapé, J., C. P. Carrino, C. Rodriguez-Penagos, O. de Gibert Bonet, C. Armentano-Oller, A. Gonzalez- Agirre, M. Melero, and M. Villegas....
  • Brown, T. B., B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss,...
  • Chang, M.-W., L. Ratinov, D. Roth, and V. Srikumar. 2008. Importance of semantic representation: Dataless classification. In Proceedings of...
  • Gao, T., A. Fisch, and D. Chen. 2021. Making pre-trained language models better few-shot learners. ACL-IJCNLP 2021 - 59th Annual Meeting of...
  • Liu, H., D. Tam, M. Muqeeth, J. Mohta, T. Huang, M. Bansal, and C. Raffel. 2022. Few-shot parameter-efficient fine-tuning is better and cheaper...
  • Liu, P., W. Yuan, J. Fu, Z. Jiang, H. Hayashi, and G. Neubig. 2021. Pre-train, prompt, and predict: A systematic survey of prompting methods...
  • Ma, T., J. G. Yao, C. Y. Lin, and T. Zhao. 2021. Issues with entailment-based zeroshot text classification. ACL-IJCNLP 2021 - 59th Annual...
  • Mahabadi, R. K., L. Zettlemoyer, J. Henderson, M. Saeidi, L. Mathias, V. Stoyanov, M. Yazdani, and M. Ai. 2022. Perfect: Prompt-free and efficient...
  • Mikolov, T., K. Chen, G. Corrado, and J. Dean. 2013. Efficient estimation of word representations in vector space. 1st International Conference...
  • Obamuyide, A. and A. Vlachos. 2018. Zeroshot relation classification as textual entailment. In Proceedings of the First Workshop on Fact Extraction...
  • Pàmies, M., J. Llop, F. Multari, N. Duran- Silva, C. Parra-Rojas, A. Gonzalez-Agirre, F. A. Massucci, and M. Villegas. 2023. A weakly supervised...
  • Phang, J., T. Févry, and S. R. Bowman. 2018. Sentence encoders on stilts: Supplementary training on intermediate labeleddata tasks.
  • Sainz, O., O. L. de Lacalle, G. Labaka, A. Barrena, and E. Agirre. 2021. Label verbalization and entailment for effective zeroand few-shot...
  • Sainz, O., I. Gonzalez-Dios, O. Lopez de Lacalle, B. Min, and E. Agirre. 2022a. Textual entailment for event argument extraction: Zero- and...
  • Sainz, O., H. Qiu, O. L. de Lacalle, E. Agirre, and B. Min. 2022b. Zs4ie: A toolkit for zero-shot information extraction with simple verbalizations....
  • Sainz, O. and G. Rigau. 2021. Ask2transformers: Zero-shot domain labelling with pre-trained language models. GWC 2021 - Proceedings of the...
  • Schick, T. and H. Schütze. 2021a. Exploiting cloze questions for few shot text classification and natural language inference. EACL 2021 -...
  • Schick, T. and H. Schütze. 2021b. It’s not just size that matters: Small language models are also few-shot learners. NAACL-HLT 2021 - 2021...
  • Tam, D., R. R. Menon, M. Bansal, S. Srivastava, and C. Raffel. 2021. Improving and simplifying pattern exploiting training. EMNLP 2021 - 2021...
  • Tunstall, L., N. Reimers, U. E. S. Jo, L. Bates, D. Korat, M. Wasserblat, O. Pereg, and H. Face. 2022. Efficient few-shot learning without...
  • Wang, S., H. Fang, M. Khabsa, H. Mao, and H. Ma. 2021. Entailment as few-shot learner. 4.
  • Xie, Q., Z. Dai, E. Hovy, M. T. Luong, and Q. V. Le. 2019. Unsupervised data augmentation for consistency training. Advances in Neural Information...
  • Yin, W., J. Hay, and D. Roth. 2019. Benchmarking zero-shot text classification: Datasets, evaluation and entailment approach. EMNLP-IJCNLP...
  • Zhang, T., F.Wu, A. Katiyar, K. Q.Weinberger, and Y. Artzi. 2020. Revisiting fewsample BERT fine-tuning.