To What Extent Is LLM Performance on Multiple-Choice Questions Driven by Data Leakage? A Case Study with Contamination-Controlled Spanish Undergraduate Exams
-
Sánchez Salido, Eva [1] ; Ghajari, Adrian [1] ; Marco, Guillermo [1] ; Gonzalo, Julio [1] ; Abizanda, Jesús [1] ; Morante, Roser [1] ; Benito-Santos, Alejandro [1] ; Plaza, Laura [1] ; Carrillo-de-Albornoz, Jorge [1] ; Fresno, Víctor [1] ; Amigó, Enrique [1] ; Fernández García, Andrés [1]
-
[1] UNED, Spain
-
- Localización: Inteligencia artificial: Revista Iberoamericana de Inteligencia Artificial, ISSN-e 1988-3064, ISSN 1137-3601, Vol. 29, Nº. 77, 2026 (Ejemplar dedicado a: Inteligencia Artificial (June 2026)), págs. 131-151
- Idioma: inglés
- DOI: 10.4114/intartif.vol29iss77pp131-151
- Enlaces
-
Resumen
The performance of Large Language Models (LLMs) on multiple-choice university-level exam benchmarks such as MMLU is often reported as highly competitive; however, such results raise persistent concerns regarding contamination of public datasets, English-centric bias, and over-reliance on aggregate accuracy as the primary evaluation signal. In particular, the widespread public availability of evaluation data makes it difficult to disentangle genuine generalization from memorization of seen content, while offering limited insight into models’ abilities on culturally grounded assessments beyond English. To address these issues, we introduce lunes (Leakage-controlled Undergraduate National Exams of Spain), a new benchmark of 11,881 multiple-choice questions drawn from official final-year undergraduate exams in Spanish, covering 104 courses across 22 degree programs. The dataset has been rigorously verified to exhibit minimal public web exposure through a combination of automated web search and manual inspection, which enables evaluation under minimal contamination conditions in a non-English, country-specific academic setting. Our results show that (i) LLMs retain strong performance on general knowledge and factual questions, even in the absence of web-accessible training data, suggesting that contamination alone does not explain their success on public benchmarks; (ii) however, their performance degrades substantially on culturally grounded and country-specific content, particularly in domains such as Spanish law, economy, and social structure. Remarkably, models consistently perform better on Anglo-centric content than on Spain-specific material even when answering in Spanish, suggesting that the bottleneck lies in culturally grounded knowledge rather than in language skills per se. A question-level error analysis further reveals that these failures reflect systematic gaps in local institutional, legal, and geographical knowledge, even for high-resource languages such as Spanish, that aggregate metrics systematically obscure.
-
Referencias bibliográficas
- Simone Balloccu, Patrícia Schmidtová, Mateusz Lango, and Ondrej Dusek. Leak, cheat, repeat: Datacontamination and evaluation malpractices...
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, et al. Language models are few-shotlearners. In H. Larochelle, M. Ranzato, R. Hadsell,...
- Center for AI Safety, Scale AI, and HLE Contributors Consortium. A benchmark of expert-levelacademic questions to assess AI capabilities....
- Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, et al. Palm: scaling language modeling withpathways. J. Mach. Learn. Res., 24(1), January...
- Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, andOyvind Tafjord. Think you have solved question...
- María Grandury, Javier Aula-Blasco, Júlia Falcão, Clémentine Fourrier, Miguel González Saiz, Gon-zalo Martínez, Gonzalo Santamaria Gomez,...
- Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and JacobSteinhardt. Measuring Massive Multitask Language...
- Minhao Jiang, Ken Liu, Ming Zhong, Rylan Schaeffer, Siru Ouyang, Jiawei Han, and Sanmi Koyejo.Does data contamination make a difference? insights...
- Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. RACE: Large-scale ReAdingComprehension Dataset From Examinations. In Proceedings...
- Cheng Li, Mengzhuo Chen, Jindong Wang, Sunayana Sitaram, and Xing Xie. Culturellm: Incor-porating cultural differences into large language...
- Yucheng Li, Yunhao Guo, Frank Guerin, and Chenghua Lin. An open-source data contaminationreport for large language models. In Yaser Al-Onaizan,...
- Percy Liang, Rishi Bommasani, Tony Lee, et al. Holistic evaluation of language models. Transactionson Machine Learning Research, 2023. Featured...
- Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conductelectricity? a new dataset for open book question...
- John X. Morris, Chawin Sitawarin, Chuan Guo, Narine Kokhlikyan, G. Edward Suh, Alexander M.Rush, Kamalika Chaudhuri, and Saeed Mahloujifar....
- Junho Myung, Nayeon Lee, Yi Zhou, Jiho Jin, Rifki Afina Putri, Dimosthenis Antypas, HsuvasBorkakoty, Eunsu Kim, Carla Perez-Almendros, Abinew...
- Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using Siamese BERT-Networks. In Proceedings of the 2019 EMNLP-IJCNLP,...
- David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, JulienDirani, Julian Michael, and Samuel R. Bowman. GPQA:...
- Oscar Sainz, Jon Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and EnekoAgirre. NLP evaluation in trouble: On the need...
- Eva Sánchez Salido, Roser Morante, Julio Gonzalo, Guillermo Marco, Jorge Carrillo-de Albornoz,Laura Plaza, Enrique Amigo, Andrés Fernandez...
- Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, AdamFisch, Adam R. Brown, Adam Santoro, Aditya Gupta,...
- Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung,Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou,...
- Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill,Omer Levy, and Samuel R. Bowman. Superglue: a stickier...
- Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman.GLUE: A multi-task benchmark and analysis platform...
- Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, WeimingRen, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li,...
- Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo-gatama, Maarten Bosma, Denny Zhou, Donald Metzler,...
- Wenxuan Zhang, Sharifah Mahani Aljunied, Chang Gao, Yew Ken Chia, and Lidong Bing. M3exam:a multilingual, multimodal, multilevel benchmark...
- Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied,Weizhu Chen, and Nan Duan. AGIEval: A human-centric...
