Bayesian Multi-Faceted TRI Models for Measuring Professor's Performance in the Classroom

• Autores: Karen Rosana Cordoba, Alvaro Mauricio Montenegro
• Localización: Revista Colombiana de Estadística, ISSN-e 2389-8976, ISSN 0120-1751, Vol. 44, Nº. 2, 2021, págs. 385-412
• Idioma: inglés
• DOI: 10.15446/rce.v44n2.89661
• Títulos paralelos:
  • Un modelo TRI de múltiples facetas bayesiano para la evaluación del desempeño docente en el aula
• Enlaces
  • Texto completo
• Resumen
  • español

    Resumen Las evaluaciones del desempeño del profesor se basan en el supuesto de que los estudiantes aprenden más de profesores altamente calificados y el hecho de que los estudiantes observan el desempeño del profesor en el aula. Sin embargo, muchos estudios cuestionan las metodologías utilizadas para tales mediciones, en general, porque los promedios de las respuestas categóricas tienen poco sentido estadístico. En este artículo, proponemos modelos Bayesianos de Teoría de Respuesta al Ítem de múltiples facetas para medir el desempeño. El modelo propuesto tiene en cuenta los efectos asociados con la severidad de los estudiantes que responden a la encuesta y los cursos que se evalúan. El modelo se aplica a un conjunto de datos obtenido de una encuesta de percepción del desempeño del profesor realizada por la Facultad de Ciencias de la Universidad Nacional de Colombia a sus estudiantes. Los puntajes del profesor que se obtienen como resultados del modelo son valores numéricos reales que se pueden usar para calcular estadísticas comunes en la evaluación del profesor. En este caso, las estadísticas son matemáticamente consistentes. Se muestra que algunos de ellos ilustran la utilidad del modelo.

  • English

    Abstract Evaluations of professor performance are based on the assumption that students learn more from highly quali-ed professors and the fact that students observe professor performance in the classroom. However, many studies question the methodologies used for such measurements, in general, because the averages of categorical responses make little statistical sense. In this paper, we propose Bayesian multi-faceted item response theory models to measure teaching performance. The basic model takes into account efects associated with the severity of the students responding to the survey, and the courses that are evaluated. The basic model proposed in this work is applied to a data set obtained from a survey of perception of professor performance conducted by Science Faculty of the Universidad Nacional de Colombia to its students. Professor scores that are obtained as model outputs are real numerical values that can be used to calculate common statistics in profesor evaluation. In this case, the statistics are mathematically consistent. Some of them are shown to illustrate the usefulness of the model.

• Referencias bibliográficas
  • Abrami, P. C.,Perry, R. P.,Leventhal, L. (1982). 'The relationship between student personality characteristics, teacher ratings, and student...
  • Ariyo, O.,Quintero, A.,Muñoz, J.,Verbeke, G.,Lesare, E. (2020). 'Bayesian model selection in linear mixed models for longitudinal data'....
  • Baker, F. B.,Kim, S.-H. (2004). Item response theory: Parameter estimation techniques. CRC Press.
  • Barkaoui, K. (2013). 'Multifaceted rasch analysis for test evaluation'. The companion to language assessment. 1301
  • Bartholomew, D. J.,Knott, M.,Moustaki, I. (2011). Latent variable models and factor analysis: A unified approach. John Wiley & Sons.
  • Basow, S. A.,Silberg, N. T. (1987). 'Student evaluations of college professors: Are female and male professors rated diferently?. Journal...
  • Becker, W. E.,Watts, M. (1999). 'How departments of economics evaluate teaching'. American Economic Review. 89. 344
  • Bélanger, C. H.,Longden, B. (2009). 'The e-ective teacher's characteristics as perceived by students'. Tertiary Education and...
  • Birnbaum, A. (1968). Statistical Theories of mental test Scores. Addison Wesley, chapter Trait models and their use in infering an examinee's...
  • Bock, R. D. (1997). 'A brief history of item response theory'. Educational Measurement: Issues and Practice. 16. 21
  • Box, G. E. (1980). 'Sampling and Bayes' inference in scienti-c modelling and robustness'. Journal of the Royal Statistical Society:...
  • Braga, M.,Paccagnella, M.,Pellizzari, M. (2014). 'Evaluating students' evaluations of professors'. Economics of Education Review....
  • Cameletti, M.,Caviezel, V. (2011). The Cronbach-Mesbah Curve for assessing the unidimensionality oh an item set: the R package CMC, in 'International...
  • Centra, J. A. (1993). Re-ective Faculty Evaluation: Enhancing Teaching and Determining Faculty E-ectiveness. The Jossey-Bass Higher and Adult...
  • Centra, J. A.,Creech, F. R. (1976). The relationship between student teachers and course characteristics and student ratings of teacher e-ectieness,...
  • Cohen, P. A.. (1981). 'Student ratings of instruction and student achievement: A meta-analysis of multisection validity studies'....
  • Cordoba, K. (2020). Un modelo TRI de múltiples facetas para la evaluación del desempeño docente en el aula. Universidad Nacional de Colombia....
  • Cronbach, L. J.. (1951). 'Coeficient alpha and the internal structure of tests',. Psychometrika. 297-334
  • Eckes, T. (2011). 'Introduction to many-facet rasch measurement'. Peter Lang. Franfurt am Main.
  • Engelhard, G. (2002). Monitoring raters in performance assessment. Erlbaum. Mahwah, NJ.
  • Engelhard, G. (2013). Invariant measurement: Using Rasch models in the social, behavioral, and health sciences. Routledge. New York, NY.
  • Feldman, K. A. (1977). 'Consistency and variability among college students in rating their teachers and courses: A review and analysis....
  • Feldman, K. A. (1978). 'Course characteristics and college students' ratings of their teachers: What we know and what we don't'....
  • Feldman, K. A. (1979). 'The significance of circumstances for college students' ratings of their teachers and courses'. Research...
  • Feldman, K. A. (1983). 'Seniority and experience of college teachers as related to evaluations they receive from students'. Research...
  • Feldman, K. A. (1987). 'Research productivity and scholarly accomplishment of college teachers as related to their instructional e-ectiveness:...
  • Feldman, K. A. (1989). 'The association between student ratings of specific instructional dimensions and student achievement: Refining...
  • Gelfand, A. E.,Dey, D. K.,Chang, H. (1992). Model determination using predictive distributions with implementation via sampling-based methods,...
  • Gelman, A.,Hwang, J.,Vehtari, A. (2014). 'Understanding predictive information criteria for bayesian models'. Statistics and computing....
  • Gelman, A.,Meng, X.-L.,Stern, H. (1996). 'Posterior predictive assessment of model fitness via realized discrepancies',. Statistica...
  • Gelman, A.. (2006). 'Prior distributions for variance parameters in hierarchical models (comment on article by browne and draper)',....
  • Homan, M. D.,Gelman, A. (2014). 'The no-u-turn sampler: adaptively setting path lengths in hamiltonian monte carlo'. Journal of Machine...
  • Jolli-e, I. (2003). 'Principal component analysis'. Technometrics. 45. 276
  • Koushki, P. A.,Kunh, H. A. J. (1982). 'How realiable are student evaluations of teachers?'. Engineering Education. 362
  • Linacre, J. M. (1989). Many-facet Rasch measurement. MESA Press. Chicago.
  • Lord, F. M.,Novick, M. R. (2008). Statistical theories of mental test scores. IAP.
  • Luo, Y.,Jiao, H. (2018). 'Using the stan program for bayesian item response theory'. Educational and psychological measurement. 78....
  • Marsh, H. W. (1987). 'Students' evaluations of university teaching: Research findings, methodological issues, and directions for future...
  • Marsh, H. W. (2007). Students' evaluations of university teaching: Dimensionality, reliability, validity, potential biases and usefulness,...
  • Martin, E. (1984). 'Power and authority in the classroom: Sexist stereotypes in teaching evaluations'. Signs: Journal of Women in...
  • Murray, H. G. (2005). Student evaluation of teaching: Has it made a difference. 'Annual Meeting of the Society for Teaching and Learning...
  • Neal, R. (2011). MCMC using Hamiltonian dynamics in Handbook of Markov Chain Monte Carlo. CRC Press. New York, NY.
  • Perry, R. P.,Niemi, R. R.,Jones, K. (1974). 'E-ect of prior teaching evaluations and lecture presentation on ratings of teaching performance'....
  • Philip, B. S.,Richard, F. (2014). 'An evaluation of course evaluations', ScienceOpen Research.
  • Small, A. C.,Hollenbeck, A. R.,Haley, R. L. (1982). 'The e-ect of emotional state on student ratings of instructors'. Teaching of...
  • Spencer, P. A.,Flyr, M. L. (1992). 'The formal evaluation as an impetus to classroom change: Myth or reality?'.
  • (2020). Stan Development Team. 'RStan: the R interface to Stan'. R package version 2.19.3.
  • (2020). Stan Development Team. 'Stan language reference manual'. Version 2.22.
  • (2020). Stan Development Team. 'Stan user's guide'. Version 2.22.
  • Uttl, B.,Eche, A.,Fast, O.,Mathison, B.,Valladares Montemayor, H.,Raab, V. (2012). 'Student evaluation of instruction/teaching (sei/set)...
  • Uttl, B.,White, C. A.,Gonzalez, D. W. (2017). 'Meta-analysis of faculty's teaching e-ectiveness: Student evaluation of teaching ratings...
  • Vehtari, A.,Gelman, A.,Gabry, J.. (2017). 'Practical bayesian model evaluation using leave-one-out cross-validation and waic'. Statistics...
  • Wachtel, H. K. (1998). 'Student evaluation of college teaching effectiveness: A brief review',. Assessment & Evaluation in Higher...
  • Watanabe, S. (2010). 'Asymptotic equivalence of bayes cross validation and widely applicable information criterion in singular learning...