شناسایی کارکرد افتراقی در آزمون پرلز ۲۰۱۶ با استفاده از درخت راش

نوع مقاله : مقاله پژوهشی

نویسندگان

1 دانش آموخته کارشناسی ارشد رشته روانسنجی، دانشگاه آزاد اسلامی، تهران، ایران

2 دانشیار گروه سنجش و اندازه‌گیری، دانشگاه علامه طباطبایی، تهران، ایران

10.22034/emes.2023.555049.2389

چکیده

هدف: دغدغۀ رو به افزایشی که در ناعادلانه بودن آزمون‌ها شکل گرفته، امکان وجود کارکرد افتراقی است. چرا که کارکرد افتراقی یا سوگیری باعث تردید در روایی آزمون می‌شود. در مطالعۀ حاضر شناسایی کارکرد افتراقی سوال‌های سواد خواندن پرلز 2016 بین دانش‌آموزان ایرانی از طریق مدل درخت راش بررسی شده است.  
روش پژوهش: داده‌های استفاده شده در این پژوهش، نتایج مطالعه بین‌المللی پرلز 2016 در ایران است که با نمونه‌گیری دو مرحله‌ای تصادفی، تعداد 4385 نفر دانش‌آموز ایرانی شامل 2143 دانش‌آموز دختر (87/48 درصد) و 2242 دانش‌آموز پسر (13/51 درصد) پایه چهارم با میانگین سنی 14/10 برای شرکت در مطالعه انتخاب شدند و در آزمون شرکت کردند. داده‏های این مطالعه در محیط  R (تحت کتابخانه psychotree) تحلیل و بررسی شد. 
یافته‌ها: یافته‌ها نشان می‌دهند  که از 181 سوال در این مطالعه بر اساس جنسیت،  بر اساس مدل درخت راش 1 سوال در میان سوالات دو ارزشی با کارکرد افتراقی یکنواخت شناسایی شد. به علاوه در بین سوالات چندارزشی، برای بلوک ۱۶ کارکرد افتراقی شناسایی شد.
نتیجه‌گیری: به نظر می‌رسد با وجود کارکرد افتراقی تاثیر آن قابل صرف‌نظر است. اگرچه بهتر است که سوال‌های با کارکرد افتراقی هنگام انجام تحلیل‌ها کنار گذاشته شود. عملکرد ضعیف دانش‌آموزان ایرانی می‌تواند متاثر از این باشد که ترجمه متون مناسب نبوده یا دانش‌آموزان هنوز به توانایی لازم برای این سطح از خواندن دست نیافته‌اند. استفاده از نتایج آزمون پرلز نیازمند ظرافت و ملاحظات خاص است.

کلیدواژه‌ها


عنوان مقاله [English]

Identifying Differential Item Functioning in PIRLS 2016: A Rasch-Tree Based Approach

نویسندگان [English]

  • Elham Pirmoradian 1
  • Asghar Minaei 2
1 Msc. Graduate, Islamic Azad University, Tehran, Iran
2 Associate Professor, Educational and Psychological Measurement, Allameh Tabataba'i University, Tehran, Iran
چکیده [English]

Objective: A growing concern in unfairness of educational assessments is the possible presence of differential item functioning. Differential item functioning or bias will undermine the validity of the assessment. In this study, identification of differential item functioning in PIRLS 2016 test among Iranian students has been considered via the Rasch-Tree model.
Methods: The data used in this research are from PIRLS 2016 exam results that was also held in Iran. A total of 4385 fourth grade Iranian students were involved consisting of 2143 girls and 2242 boys with the average age of 10.14. The analysis performed by the authors on this dataset was written in R programming language.
Results: The results indicate that out of 181 questions, based on raschtree model, one dichotomous question shows uniform differential item functioning. Moreover, among polytomous questions, block 16 shows differential item functioning. 
Conclusion: It appears even though there is differential item functioning in the PIRLS 2016 test for Iranian students, the impact is negligible. Ideally, however, we would want to remove the questions with differential item functioning before conducting any analysis. Using test results requires care and discretion.

کلیدواژه‌ها [English]

  • Keywords: Differential Item Functioning
  • PIRLS 2016
  • Rasch-tree model

References

Ahmadi Deh Qutbuddini M., Khodai E., Farzad V., Moghadam-Zadeh A., & Kabiri M.(2017), Applying Bi-factor Multidimensional Item-response Theory Model for Dimensionality and Differential items Functioning Analysis on Testlet-Based Tests. Quarterly of Educational Measurement, 7(28), 121-153.
Alimirzaie M., Moghadam-Zadeh A., Minaei A., Ezanloo B. & Salehi K. (2019). Sources of Question Differential Function and its Application in Education. Journal of Research in Teaching, 7(1), 133-153.
Alshehri, Y. M. (2020). Empirical Evaluation for Gender-Based DIF in an International Assessment: The Case of PIRLS (Doctoral dissertation, University of Kansas).
American Educational Research Association. (2018). Standards for educational and psychological testing. American Educational Research Association.
Anderson, R. C., & Pearson, P. D. (1984). A Schema-Theoretic View of Basic Processes in Reading Comprehension. Technical Report No. 306.
Anderson, R. C., & Pearson, P. D. (1988). A schema-theoretic view of basic processes in reading comprehension. Interactive approaches to second language reading, 37-55.
Asgari, A. (2007). New Approaches to Psychometrics Part Three: Models of Item Response Theory, Rasch Models. Journal of Iranian Psychologists, 4(13), 80-83.
Barro, R. J., & Lee, J. W. (2001). International data on educational attainment: updates and implications. Oxford Economic papers, 53(3), 541-563.
Bauer, D. J. (2017). A more general model for testing measurement invariance and differential item functioning. Psychological methods, 22(3), 507.
Bollmann, S., Berger, M., & Tutz, G. (2017). Item-Focused Trees for the Detection of Differential Item Functioning in Partial Credit Models. Educational and Psychological Measurement, 1-24.
Coulombe, S., Tremblay, J. F., & Marchand, S. (2004). Literacy scores, human capital and growth across fourteen OECD countries (Vol. 89). Geological Survey of Canada.
Desjardins, C. D., & Bulut, O. (2018). Handbook of educational measurement and psychometrics using R. CRC Press.
Finch, W. H., & French, B. F. (2007). Detection of crossing differential item functioning: A comparison of four methods. Educational and psychological Measurement, 67(4), 565-582.
Finch, W. H., Hernández Finch, M. E., & French, B. F. (2016). Recursive partitioning to identify potential causes of differential item functioning in cross-national data. International Journal of Testing, 16(1), 21-53.
Fischer, G. H., & Molenaar, I. W. & (Eds.). (1995). Rasch Models: Foundations، Recent Developments and Applications. New York: Springer-Verlag.
Frick, H., Strobl, C., & Zeileis, A. (2015). Rasch mixture models for DIF detection: A comparison of old and new score specifications. Educational and Psychological Measurement, 75(2), 208-234.
Geske, A., & Ozola, A. (2010). Differential item functioning in the aspect of gender differences in reading literacy. In The 4th IEA International Research Conference.
Giraud, F. (2007). Christian Baudelot, Roger Establet, Quoi de neuf chez les filles? Entre stéréotypes et libertés. Lectures.
Gustafsson, J. E. (1980). Testing and obtaining fit of data to the Rasch model. British Journal of Mathematical and Statistical Psychology, 33(2), 205-233.
Hambleton, R. K., Swaminathan, H., & Rogers, H. J. (1991). Fundamentals of item response theory (Vol. 2). Sage.
Jeffers, H. (2020). Using rasch tree to detect uniform differential item functioning and item difficulty parameters in the progress in international reading literacy data (PIRLS). Ball State University.
Kabiri M., Karimi A., & Bakhshali-Zadeh S. (2017).  PIRLS 2016 findings, Madreseh Publications, Tehran, Iran.
Kamata, A., & Vaughn, B. K. (2004). An Introduction to Differential Item Functioning Analysis. Learning Disabilities: A Contemporary Journal, 2(2), 49-69.
Karami, H., & Salmani Nodoushan, M. A. (2011). Differential Item Functioning (DIF): Current Problems and Future Directions. International Journal of Language Studies (IJLS), 5(3), 133-142.
Kennedy, A. M. (2008). Examining gender and fourth graders’ reading habits and attitudes in PIRLS 2001 and 2006. In 3rd IEA International Research Conference, Taipei, Chinese Taipei.
Millsap, R. E. (2012). Statistical approaches to measurement invariance. Routledge.
Minaie A. (2015). Aapplication of Rasch measurement model to evaluate measurement properties of the Test of Visual- Motor Skills-Revised, Educational Measurement, 5(18), 77-114
Minaei A. & Ghafari Z., (2015) Differential Items functioning of Mathematics Test of TIMSS 2011 in Grade 8 between Girls and Boys using Item-Response Theory Approach (IRT). Educational Measurement, 6(21), 21-40
Mullis, I. V. S., Martin, M. O., Foy, P., & Hooper, M. (2017). PIRLS 2016 International Results in Reading. Retrieved from Boston College, TIMSS & PIRLS International Study Center website: http://timssandpirls.bc.edu/pirls2016/international-results/
Mullis, I. V., Kennedy, A. M., Martin, M. O., & Sainsbury, M. (2006). Assessment Framework and Specifications. PIRLS 2006. International Association for the Evaluation of Educational Achievement. Herengracht 487, Amsterdam, 1017 BT, The Netherlands.
Roux, K. (2020). Examining the equivalence of the PIRLS 2016 released texts in South Africa across three languages (Doctoral dissertation, University of Pretoria).
Sari, I. M. (2020). Decomposing Gender Differences in Reading: Evidence from Nordic PIRLS 2016 (Master's thesis).
Schulz-Heidorf, K., & Støle, H. (2018). Gender differences in Norwegian PIRLS 2016 and ePIRLS 2016 results at test mode, text and item format level. Nordic Journal of Literacy Research, 4(1).
Strobl, C., Kopf, J., & Zeileis, A. (2011a). A new method for detecting differential item functioning in the Rasch model. Innsbruck: University of Innsbruck Working Papers in Economic and Statistic.
Strobl, C., Kopf, J., & Zeileis, A. (2011b) Using the raschtree function for detecting differential item functioning in the Rasch model.‏ Innsbruck: University of Innsbruck Working Papers in Economic and Statistic.
Strobl, C., Kopf, J., & Zeileis, A. (2015). Rasch trees: A new method for detecting differential item functioning in the Rasch model. Psychometrika, 80(2), 289-316.
Stubbe, T. C. (2011). How do different versions of a test instrument function in a single language? A DIF analysis of the PIRLS 2006 German assessments. Educational Research and Evaluation, 17(6), 465-481.
Thissen, D., Steinberg, L., & Wainer, H. (1993). Detection of differential item functioning using the parameters of item response models.‏
Westers, P., & Kelderman, H. (1992). Examining differential item functioning due to item difficulty and alternative attractiveness. Psychometrika, 57(1), 107-118.
Zeileis, A., Strobl, C., Wickelmaier, F., & Kopf, J. (2010). psychotree: Recursive partitioning based on psychometric models. R package version 0.11-1, URL http://CRAN. R-project. org/package= psychotree. University of Innsbruck-Working Papers in Economics and Statistics Recent Papers.
Zumbo, B. D. (1999). A handbook on the theory and methods of differential item functioning (DIF). Ottawa: National Defense Headquarters, 160.