Validity of PhD Entrance Exams of Engineering in 1397

Document Type : Original Article

Authors

10.22034/emes.2020.44374

Abstract

The quality of a PhD candidate’s performance is the most important factor that can affect admission committee members’ decision-making and selection of applicants. One of the issues that has always accompanied candidates’ scores  in admission exams is the validity of these scores in predicting the candidates’ successful performance . The primary purpose of this research was evaluating the predictive criterion validity of subject tests scores  in four engineering PhDprograms on interview measures  as the criterion for successful performance . In this descriptive correlational study, candidates’ scores in subject tests  and interviews  in four engineering PhD programs in 2018 were analyzed based on the Generalizability theory. Depending on their majors, examinees’ data included their scores on 3 to 8 subtests in subject tests and 2 composite interview criteria . Data were analyzed using mGENOVA software based on a multivariate analysis with a single facet  design. The relationship between two composite universe scores was illustrated by the bivariate normal distribution in a typical scatter diagram. The results show that subject tests scores can predict candidates’ scores on the interview criterion as an index for successful performance except in the field of Information Technology. The large number of subtests in subject tests,their instability, and the construct irrelevant variance were among the factors that may influence the accuracy of predictions, especially on the cut scores, for accepting or rejecting candidates. Accurately predicting the success of an applicant’s performance  based on test scores   is an unattainable goal. However, precisely defining the criteria for success  in PhD programs, increasing the significance of general English and aptitude tests, adjusting the difficulty level of tests, reducing measurement error in predictive measures  , having a two-step assessment program in which the general language and aptitude tests are administered as a primary screening by NOET and the subject tests and interviews by universities can be effective method to increase the criterion validity of test scores in predicting candidates’ successful performance and enhance the chances of admitting more qualified students.

Keywords

References

آلن، مری جی. و ین، وندی ام. (1979). مقدمه‌ای بر نظریه‌های اندازه‌گیری (روان‌سنجی)؛ ترجمه علی دلاور (1374). تهران: انتشارات سازمان مطالعه و تدوین کتب علوم انسانی دانشگاه‌ها (سمت).
ذوالفقارنسب، سلیمان؛ خدایی، ابراهیم؛ یادگارزاده، غلامرضا (1391). وزن‌دهی بهینه به سؤال­ها و خرده‌آزمون‌های ورودی برای ساخت نمره کل ترکیبی. فصلنامه مطالعات اندازه‌گیری و ارزشیابی آموزشی، 3(4)، 79- 104.
ذوالفقارنسب، سلیمان؛ یادگارزاده، غلامرضا (1391) چهارچوب سامانه نوآوری و کارکرد آن در رشد و توسعه ملی: رویکردی بر پایه نشانگرهای سازمان ملل متحد. رهیافت، 51، 41-50.
فرجی ده‌سرخی، حاتم؛ آراسته، حمیدرضا؛ نوه‌ابراهیم، عبدالرحیم؛ عبداللهی، بیژن (1394) شناسایی ملاک‌ پذیرش دانشجویان دوره دکتری: مطالعه کیفی. فصلنامه پژوهش و برنامه‌ریزی در آموزش عالی، 21(4)، 97-71.
فرجی ده‌سرخی، حاتم؛ آراسته، حمیدرضا؛ بغدادچی، رضا؛ نوه‌ابراهیم، عبدالرحیم؛ عبدالهی، بیژن (1395) سیستم پذیرش دکتری در ایران و دانشگاه‌های برتر: یک مطالعه تطبیقی. مجله علوم تربیتی دانشگاه شهید چمران اهواز، 6(2)، 5-34.
وزارت بهداشت، درمان، و آموزش پزشکی (1397).آیین‌نامه آموزشی دوره دکتری تخصصی(Ph.D.). مصوب شصت و نهمین جلسه شورای عالی برنامه ریزی علوم پزشکی: http://satim.tums.ac.ir/app/webroot/upload/files/PhD(2).pd
American Educational Research Association. American Psychological Association & National Council for Measurement in Education [AERA, APA & NCME] (2014). The Standards for Educational and Psychological Testing. Washington.
Attiyeh, G. M. (1999). Determinants of persistence of graduate students in Ph. D. programs. ETS Research Report Series, 1999(1), i-43.
Bachman, L. F., & Palmer, A. (1996). Language testing in practice. Oxford, UK: Oxford University Press.
Brennan, R. L. (2003). Coefficients and Indices in Generalizability Theory. (CASMA Research Report No.1). Iowa City: Center for Advanced Studies in Measurement and Assessment, the University of Iowa.
Brennan, R. L. (2001). Generalizability Theory, Iowa Testing Programs. University of Iowa. Springerverlag, New York.
Fogarty, G. J. (2008). Principles and applications of educational and psychological testing. In Adult educational psychology (pp. 351-383). Brill Sense.
Godin, B. (2003). The emergence of S&T indicators: why did governments supplement statistics with indicators? Research Policy, 32(4), 679-691.
Hartnett, R. T., & Willingham, W. W. (1980). The criterion problem: What measure of success in graduate education? Applied Psychological Measurement, 4(3), 281-291.
Hunter, J. E., & Hunter, R. F. (1984). Validity and utility of alternative predictors of job performance. Psychological bulletin, 96(1), 72.
Kuncel, N. R., Hezlett, S. A., & Ones, D. S. (2001). A comprehensive meta-analysis of the predictive validity of the graduate record examinations: implications for graduate student selection and performance. Psychological bulletin, 127(1), 162.
Louw, J., & Muller, J. (2014). A literature review on models of the PhD. Centre for Higher Education Trust (CHET). Available online at: https://www.chet.org.za/papers/literature-review-models-phd
Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13-104). New York, NY: American Council on education and Macmillan.
Messick, S. (1993). Foundations of validity: Meaning and consequences in psychological assessment. ETS Research Report Series, 1993(2), i-18.
Otago university (2011). Selecting high quality PhD candidates seven tips for prospective supervisors. Graduate Research Services. New Zeland. Available at: https://www.otago.ac.nz/graduate-research/study/otago029265.pdf
Rogelberg, S. G. (Ed.). (2007). Encyclopedia of industrial and organizational psychology (Vol. 1). Sage.
Schmidt, F. L., Hunter, J. E., McKenzie, R. C., & Muldrow, T. W. (1979). Impact of valid selection procedures on work-force productivity. Journal of Applied Psychology, 64(6), 609.
Shaw, E. J., Marini, J. P., Beard, J., Shmueli, D., Young, L., & Ng, H. (2016). The Redesigned SAT® Pilot Predictive Validity Study: A First Look. Research Report 2016-1. College Board.
Taylor, H. C., & Russell, J. T. (1939). The relationship of validity coefficients to the practical effectiveness of tests in selection: discussion and tables. Journal of applied psychology, 23(5), 565.
Thompson, W. J. (2018). The SAGE Encyclopedia of Educational Research, Measurement, and Evaluation.
Walpole, M., Burton, N. W., Kanyi, K., & Jackenthal, A. (2002). Selecting successful graduate students: in‐depth interviews with GRE® users. ETS research report series, 2002(1), I-29.
Willingham, W. W. (1976). Validity and the Graduate Record Examinations Program.
Zwick, R. (1991). An Analysis of Graduate School Careers in Three Universities: Differences in Attainment Patterns across Academic Programs and Demographic Groups. GRE Board Report No. 86-21P
Zwick, R. (2006). Higher education admissions testing. In R. L. Brennan (Ed.), Educational Measurement (4th ed.), pp. 647-679 Westport, CT: American Council on Education/Praeger.
Zwick, R. (2007). College admission testing. National Association for College Admission Counseling, 1-44
Educational Measurement and Evaluation Studies
Volume 10, Issue 30
August 2020
Pages 135-185

Files

Share

How to cite

Statistics