Ir al contenido

Documat

Testing invariance between web and paper students satisfaction surveys: a case study

  • Monica Martinez-Gomez [1] ; Juan A. Marin-Garcia [1] ; Martha Giraldo O'Meara [2]
    1. [1] Universidad Politécnica de Valencia

      Universidad Politécnica de Valencia

      Valencia, España

    2. [2] Universitat de València

      Universitat de València

      Valencia, España

  • Localización: Intangible Capital, ISSN-e 1697-9818, Vol. 13, Nº. 5, 2017, págs. 879-901
  • Idioma: inglés
  • DOI: 10.3926/ic.1049
  • Enlaces
  • Resumen

    • Purpose: This paper studied the measurement invariance (MI) across web-based and paper-based surveys to evidece if both techniques of data collection can be regarded as equivalent.

      Design/methodology/approach: We develop a multigroup confirmatory factor analysis (MGCFA) with Maximum Likelihood Estimation to asses meassurement invariance of the Job Diagnostic Survey (JDS) adapted to teaching, with data collected from paper and web surveys. Sample from paper surveys was constituted by 294 student of a Spanish public university in the academic years 2007-08, 2008-09 and 2009-10. Internet surveys were administered through an open source survey application called LimeSurvey. We received 241 completed questionnaires.

      Findings: Results show that metric invariance, covariance invariance, variance of latent factors invariance and measurement errors invariance can be established between two groups. We can conclude that both methods of collecting data can be considered equivalent.

      Research limitations/implications: This study was done with a particular sample and strict focus questionnaire and we might not generalize the findings. It should be extended in the future to include other universities and graduate students.

      Originality/value: Results showed that the factor structures remained invariant across the internet-based and paper-based groups, that is to say, both methods of collecting data can be considered equivalent, with the same factor structure, factor loadings, measurement errors of factors and the same reliability. These findings are useful for researchers since they add a new sample in which web and paper questionnaires are equivalent and for teachers to desire to change the teaching methodology at university, encourage students’ participation and teamwork through active methodologies

  • Referencias bibliográficas
    • American Psychological Association & National Council on Measurement in Education. (1999). Standards for educational and psychological...
    • Aster, A.Z. (2004). Consumer research goes online. Marketing Magazine, 7, 13-14
    • Aydin, B., & Ceylan, A. (2008). The employee satisfaction in metalworking manufacturing: How do organizational culture and organizational...
    • Barak, M., Ben-Chaim D., & Zoller, U. (2007). Purposely teaching for the promotion of higher-order thinking skills: A case of critical...
    • Bartram, D. (2005). The great eight competencies: A criterion-centric approach to validation. Journal of Applied Psychology, 90, 1185-1203....
    • Bentler, P.M., & Bonett, D.G. (1980). Significance tests and goodness of fit in the analysis of covariance structures. Psychological Bulletin,...
    • Bollen, K.A., & Long, J.S. (1993). Testing Structural Equation Models. Newbury Park, California: Sage
    • Bosnjak, M., Tuten, T.L., & Wittmann, W.W. (2005). Unit (non) response inweb-based access panel surveys: an extended planned-behavior...
    • Bowling, A. (2005). Mode of questionnaire administration can have serious effects on data quality. Journal of Public Health, 27, 281-291....
    • Brown, M.W., & Cudeck, R. (1993). Alternative ways of assessing model fit. In K.A. Bollen & J.S. Long (Eds.), Testing structural equation...
    • Buchanan, T., & Smith, J.L. (1999). Using the internet for psychological research: Personality testing on the World Wide Web. British...
    • Byrne, B.M. (1989). Multigroup comparisons and the assumption of equivalent construct validity across groups: Methodological and substantive...
    • Byrne, B.M., & Stewart, S.M. (2006).The MACS approach to testing for multigroup invariance of a second-order structure: A walk through...
    • Byrne, B.M., & Van De Vijver, F.J.R. (2010). Testing for measurement and structural equivalence in large-scalecross-cultural studies:...
    • Chen, F.F. (2007). Sensitivity of goodness of fit indices to lack of measurement invariance. Structural Equation Modeling, 14, 464-504. https://doi.org/10.1080/10705510701301834
    • Chen, F.F.; Sousa, K.H.; West, S.G. (2005). Testing Measurement Invariance of Second-Order Factor Models. Structural Equation Modeling, 12:...
    • Cheung, G.W. (2008). Testing equivalence in the structure, means, and variances of higher-order constructs with structural equation modelin....
    • Cheung, G.W., & Rensvold, R.B. (2002). Evaluating goodness-of-fit indices for testing measurement Equivalence. Structural Equation Modeling,...
    • Cohen, S., Kamarck, T., & Mermelstein, R. (1983). A global measure of perceived stress. Journal of Health and Social Behavior, 24, 385-396....
    • Cole, M.S., Bedeian, A.G., & Feild, H.S. (2006). The measurement equivalence of web-based and paperand-pencil measures of transformation...
    • Cook, C., Heath, F., & Thompson, R.L. (2000). A meta-analysis of response rates in Web-or Internet based surveys. Educational and Psychological...
    • Curran, P.J., West, S.G., & Finch, J.F. (1996). The robustness of test statistics to nonnormality and specification error in confirmatory...
    • Davidov, E., & Depner, F. (2011). Testing for measurement equivalence of human values across online and paper-and pencil surveys. Quality...
    • De Beuckelaer, A., & Lievens, F. (2009). Measurement equivalence of paper-and-pencil and Internet organisational surveys: A large scale...
    • Dillman, D.A. (2000). Mail and Internet Surveys: The Tailored Design Method (2nd Eds.). New York: John Wiley & Sons
    • Dillman, D., Smyth, J., & Christian, L. (2009). Internet, Mail, and Mixed-Mode Surveys: The Tailored Design Method. New York: Wiley
    • Dimitrov, D.M. (2006). Validation of cognitive operations and processes across ability levels and individual test items. In T.E. Scruggs &...
    • Drasgow, F., & Schmidt, N. (2002). Measuring and Analyzing Behaviour in Organizations: Advances in Measurement and Data Analysis. San...
    • Ebenezer, J.V., Columbus, R., Kaya, O.N., Zhang, L., & Ebenezer, D.L. (2012). One science teacher's professional development experience:...
    • Elosua, P. (2005). Evaluación progresiva de la invarianza factorial entre las versiones original y adaptada de una escala de autoconcepte....
    • Epstein, J., Klinkenberg, W.D., Wiley, D., & McKinley, L. (2001). Ensuring sampling equivalence across Internet and paper-and-pencil assessments....
    • Fan, W., & Yan, Z. (2010). Factors affecting response rates of the web survey: A systematic review. Computers in Human Behavior, 26(2),...
    • Fang, J., Wen, C., & Pavur, R. (2012). Participation willingness in web surveys: Exploring effect of sponsoring corporation's and...
    • Fang, J., Wen, C., & Prybutok, V.R. (2014). An assessment of equivalence between internet and paperbased surveys: Evidence from collectivistic...
    • Fouladi, R.T., McCarthy, C.J., & Moller, N.P. (2002). Paper-and-pencil or online? Evaluating mode effects on measures of emotional functioning...
    • Gangestad, S., & Snyder, M. (1985). To carve nature at its joints': On the existence of discrete classes in personality. Psychological...
    • Giraldo-O'Meara, M., Marin-Garcia, J.A., & Martínez-Gómez, M. (2014). Validation of the JDS satisfaction scales applied to educational...
    • Göritz, A.S. (2006). Incentives in web studies: methodological issues and a review. International Journal of Internet Science, 1(1), 58-70
    • Hackman, J.R., & Oldham, G.R. (1975). Development of the Job Diagnostic Survey. Journal of Applied Psychology, 60(2), 159-170. https://doi.org/10.1037/h0076546
    • Hackman, J.R., & Oldham, G.R. (1976). Motivation through the design of the work: Test of a theory. Organizational Behaviour and Human...
    • Hackman, J.R., & Oldham, G.R. (1980). Work Redesig. Reading, MA: Addison-Wesley
    • Hair, J.F., Anderson, R.E., Thatam, R.L., & Black, W.C. (1998). Multivariate Data Analysis (6th eds). New York: Prentice Hall International
    • Herrero, J., & Meneses, J. (2006). Short Web-based versions of the perceived stress (PSS) and Center for Epidemiological Studies-Depression...
    • Hogg, A. (2003). Web efforts energize customer research. Electronic Perspectives, 6, 81-83
    • Hohwü, L., Lyshol, H., Gissler, M., Jonsson, S.H., Petzold, M., & Obel, C. (2013). Web-based versus traditional paper questionnaires:...
    • Hu, L., & Bentler, P.M. (1995). Evaluating model fit. In R. H. Hoyle (Ed.), Structural Equation Modeling. Concepts, Issues, and Applications...
    • Hu, L., & Bentler, P.M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives....
    • Ismail, A., Mashkuri, A., Sulaiman, A., & Kee Hock, W. (2011). Interactional justice as a mediator of the relationship between pay for...
    • Jöreskog, K.G. (1971). Simultaneous factor analysis in several populations. Psychometrika, 36, 409-426. https://doi.org/10.1007/BF02291366
    • King, W., & Miles, E. (1995). Quasi-experimental assessment of the effect of computerizing noncognitive paper and pencil measurements:...
    • Kline, R.B. (2010). Principles and Practice of Structural Equation Modeling. NY, London: The Guilford Press
    • Kraut, A.I., & Saari, L.M. (1999). Organizational surveys: Coming of age for a new era. In A.I. Kraut & A.K. Korman (Eds), Evolving...
    • Lautenschlager, G.J., & Flaherty, V.L. (1990). Computer administration of questions: More desirable or more social desirability?. Journal...
    • Leung, D., & Kember, D. (2005). Comparability of data gathered from evaluation questionnaires on paper and through the internet. Research...
    • Marbach-Ad, G., & Sokolove, P.G. (2002). The use of e-mail and in-class writing to facilitate student-instructor interaction in large-enrollment...
    • Martínez-Gomez, M., & Marin-Garcia, J.A. (2009). Como medir y guiar el cambio hacia entornos educativos universitarios más motivadores...
    • Martínez Gómez, M., Marin-Garcia, J., & Giraldo-O'Meara, M. (2016). The measurement invariance of job diagnostic survey (jds) across...
    • Martins, N. (2010). Measurement model equivalence in web-and paper-based surveys. Sourthen African Business Review, 14(3), 77-107
    • Meade, A.W., Michels, L.C., & Lautenschlager, G.J. (2007). Are internet and paper-and pencil personality tests truly comparable? An experimental...
    • Miles, E.W., & King, W.C. (1998). Gender and administration mode effects when pencil-and-paper personality tests are computerized. Educational...
    • Nulty, D.D. (2008). The adequacy of response rates to online and paper surveys: What can be done?. Assessment & Evaluation in Higher Education,...
    • Orgambídez-Ramos, A., Borrego-Alés, Y., & Mendoza-Sierra, I. (2014). Role stress and work engagement as antecedents of job satisfaction...
    • Radloff, L. (1977). The CES-D Scale: A self-report depression scale for research in the general population. Applied Psychological Measurement,...
    • Riva, G., Teruzzi, T., & Anolli, L. (2003). The use of the internet in psychological research: Comparison of online and offline questionnaires....
    • Reips, U.D. (2000). The web experiment method: Advantages, disadvantages and solutions. In M.H. Birnbaum (Eds.), Psychological Experiments...
    • Roberts, L.L., Konczak, L.J., & Macan, T.H. (2004). Effects of data collection method on organizational climate survey results. Applied...
    • Santos Rego, M.á., Godás Otero, A., Lorenzo Moledo, M., & Gómez Fraguela, J.A. (2010). Eficacia y satisfacción laboral de dos profesores...
    • Satorra, A., & Bentler, P.M. (1994). Corrections to test statistic and standard errors in covariance structure analysis. In A. von Eye...
    • Satorra, A., & Bentler, P.M. (2001). A Scaled difference chi-square test statistic for moment structure analysis. Psychometrika, 66(4),...
    • Schaeffer, D., & Dillman, D. (1998). Development of a standard e-mail methodology: Results of an experiment. Public Opinion Quarterly,...
    • Schonlau, M., Fricker, R.D., & Elliott, M.N. (2002). Conducting research surveys via e-mail and the web. Santa Monica, CA: Rand Corporation
    • Simsek, Z., & Veigha, J.F. (2001). A primer on internet organizational surveys. Organizational Research Methods, 4, 218-235. https://doi.org/10.1177/109442810143003
    • Sproull, L.S. (1986). Using electronic mail for data collection in organizational research. Academy of Management Journal, 29, 159-169. https://doi.org/10.2307/255867
    • Stanton, J.M. (1998). An empirical assessment of data collection using the internet. Personnel Psychology, 51, 709-725. https://doi.org/10.1111/j.1744-6570.1998.tb00259.x
    • Steenkamp, J.B.E.M., & Baumgartner, H. (1998). Assessing measurement invariance in crossnational consumer research. Journal of Consumer...
    • Steinmetz, H., Schmidt, P., Tina-Booh, A., Wieczorek, S., & Schwartz, S. (2009). Testing measurement invariance using multigroup CFA:...
    • Trullas, I., & Enache, M. (2011). Theoretical analysis of the antecedents and the consequences of students' identification with their...
    • Ullman, J.B., & Bentler, P.M. (2004) Structural Equation Modeling. In M. Hardy & Bryman (Eds.), Handbook of Data Analysis (pp.431-458)....
    • Van Gelder, M.M., Bretveld R.W., & Roeleveld, N. (2010). Web-based questionnaires: The future in epidemiology?. American Journal of Epidemiol,...
    • Vandenberg, R.J. (2002). Toward a further understanding of and improvement in measurement invariance methods and procedures. Organizational...
    • Vandenberg, R.J., & Lance, C.E. (2000). A review and synthesis on the measurement invariance literature: Suggestions, practices and recommendations...
    • Van de Schoot, R., Lugtig, P., & Hox, J. (2012). A checklist for testing measurement invariance. European Journal of Developmental Psychology,...
    • Walt, N., Atwood, K., & Mann, A. (2008). Does Survey Medium Affect Responses? An Exploration of Electronic and Paper Surveying in British...
    • Young, S.A., Daum, D.L., Robie, C., & Macey, W.H. (2000). Paper vs. web survey administration: Do different methods yield different results?....
    • Yu, S.C., & Yu, M.N. (2007). Comparison of Internet-based and paper-based questionnaires in Taiwan using multisample invariance approach....
    • Yun, G.W. & Trumbo, C.W. (2000). Comparative Response to a Survey Executed by Post, e-mail, & Web Form. Journal of Computer-Mediated...
Fundación Dialnet

Mi Documat

Opciones de artículo

Opciones de compartir

Opciones de entorno