Analysis of ChatGPT Performance in Computer Engineering Exams

• Roberto Rodriguez-Echeverría ^[1] ; Juan D. Gutiérrez ^[2] ; José M. Conejero ^[1] ; Álvaro E. Prieto ^[1]
  1. [1] Universidad de Extremadura

     Universidad de Extremadura

     Badajoz, España

     
  2. [2] Universidade de Santiago de Compostela

     Universidade de Santiago de Compostela

     Santiago de Compostela, España

     
• Localización: Revista Iberoamericana de Tecnologías del Aprendizaje: IEEE-RITA, ISSN 1932-8540, Vol. 19, Nº. 1, 2024, págs. 71-80
• Idioma: inglés
• DOI: 10.1109/RITA.2024.3381842
• Texto completo no disponible (Saber más ...)
• Resumen

  • The appearance of ChatGPT at the end of 2022 was a milestone in the field of Generative Artificial Intelligence. However, it also caused a shock in the academic world. For the first time, a simple interface allowed anyone to access a large language model and use it to generate text. These capabilities have a relevant impact on teaching-learning methodologies and assessment methods. This work aims to obtain an objective measure of ChatGPT’s possible performance in solving exams related to computer engineering. For this purpose, it has been tested with actual exams of 15 subjects of the Software Engineering branch of a Spanish university. All the questions of these exams have been extracted and adapted to a text format to obtain an answer. Furthermore, the exams have been rewritten to be corrected by the teaching staff. In light of the results, ChatGPT can achieve relevant performance in these exams; it can pass many questions and problems of different natures in multiple subjects. A detailed study of the results by typology of questions and problems is provided as a fundamental contribution, allowing recommendations to be considered in the design of assessment methods. In addition, an analysis of the impact of the non-deterministic aspect of ChatGPT on the answers to test questions is presented, and the need to use a strategy to reduce this effect for performance analysis is concluded.

• Referencias bibliográficas
  • E. Tovar, Ed., JENUI 97: III Jornadas de Enseñanza Universitaria de Informática, Madrid, 16 y 17 de junio de 1997. Madrid, Spain: Facultad...
  • J. Divasón, F. J. Martínez de Pisón, A. Romero, and E. Sáenz de Cabezón, “Modelos de inteligencia artificial para asesorar el proceso evaluador...
  • W. S. Mcculloch and W. Pitts, “A logical calculus of the ideas immanent in nervous activity,” Bull. Math. Biophys., vol. 5, no. 4, pp. 115–133,...
  • OpenAI. (Nov. 2022). ChatGPT: Optimizing Language Models for Dialogue. [Online]. Available: https://openai.com/blog/chatgpt/
  • A. Vaswani et al., “Attention is all you need,” in Proc. Adv. Neural Inf. Process. Syst., vol. 30, 2017, pp. 1–11.
  • T. B. Brown et al., “Language models are few-shot learners,” in Proc. Adv. Neural Inf. Process. Syst., vol. 33, 2020, pp. 1–25.
  • S. Zhang et al., “OPT: Open pre-trained transformer language models,” 2022, arXiv:2205.01068.
  • B. Workshop et al., “Bloom: A 176B-parameter open-access multilingual language model,” 2022, arXiv:2211.05100.
  • A. Gilson et al., “How does ChatGPT perform on the medical licensing exams? The implications of large language models for medical education...
  • M. J. Bommarito and D. M. Katz, “GPT takes the bar exam,” 2022. [Online]. Available: https://ssrn.com/abstract=4314839
  • S. Wollny, J. Schneider, D. Di Mitri, J. Weidlich, M. Rittberger, and H. Drachsler, “Are we there yet?—A systematic literature review on...
  • X. Zhai. (2022). ChatGPT User Experience: Implications for Education. [Online]. Available: https://ssrn.com/abstract=4312418
  • M. M. Alshater, “Exploring the role of artificial intelligence in enhancing academic performance: A case study of ChatGPT,” 2022, arXiv:2212.14402....
  • T. Susnjak, “ChatGPT: The end of online exam integrity?” 2022, arXiv:2212.09292.
  • D. R. E. Cotton, P. A. Cotton, and J. R. Shipway, “Chatting and cheating: Ensuring academic integrity in the era of ChatGPT,” Innov. Educ....
  • Y. K. Dwivedi et al., “Opinion Paper: ‘So what if ChatGPT wrote it?’ Multidisciplinary perspectives on opportunities, challenges and implications...
  • F. J. García-Peñalvo, F. Llorens-Largo, and J. Vidal, “La nueva realidad de la educación ante los avances de la inteligencia artificial generativa,”...
  • R. Rodriguez-Echeverria, J. D. Gutiérrez, J. M. Conejero, and A. E. Prieto, “Impacto de ChatGPT en los métodos de evaluación de un grado...
  • K. Mahowald, A. A. Ivanova, I. A. Blank, N. Kanwisher, J. B. Tenenbaum, and E. Fedorenko, “Dissociating language and thought in large language...
  • N. Kassner and H. Schütze, “Negated and misprimed probes for pretrained language models: Birds can talk, but cannot fly,” in Proc. 58th Annu....
  • A. M. Turing, “I. Computing machinery and intelligence,” Mind, vol. 59, no. 236, pp. 433–460, Oct. 1950, doi: 10.1093/mind/LIX.236.433.
  • L. Lidén, “Artificial stupidity: The art of intentional mistakes,” AI Game Program. Wisdom, vol. 2, no. 5, pp. 41–48, 2003.
  • G. O. Young, “Synthetic structure of industrial plastics (Book style with paper title and editor),” in Plastics, vol. 3, 2nd ed., J. Peters,...