Identifying Differential Item Functioning in PIRLS 2016: A Rasch-Tree Based Approach

Document Type : Original Article

Authors

1 Msc. Graduate, Islamic Azad University, Tehran, Iran

2 Associate Professor, Educational and Psychological Measurement, Allameh Tabataba'i University, Tehran, Iran

10.22034/emes.2023.555049.2389

Abstract

Objective: A growing concern in unfairness of educational assessments is the possible presence of differential item functioning. Differential item functioning or bias will undermine the validity of the assessment. In this study, identification of differential item functioning in PIRLS 2016 test among Iranian students has been considered via the Rasch-Tree model.
Methods: The data used in this research are from PIRLS 2016 exam results that was also held in Iran. A total of 4385 fourth grade Iranian students were involved consisting of 2143 girls and 2242 boys with the average age of 10.14. The analysis performed by the authors on this dataset was written in R programming language.
Results: The results indicate that out of 181 questions, based on raschtree model, one dichotomous question shows uniform differential item functioning. Moreover, among polytomous questions, block 16 shows differential item functioning. 
Conclusion: It appears even though there is differential item functioning in the PIRLS 2016 test for Iranian students, the impact is negligible. Ideally, however, we would want to remove the questions with differential item functioning before conducting any analysis. Using test results requires care and discretion.

Keywords


References

Ahmadi Deh Qutbuddini M., Khodai E., Farzad V., Moghadam-Zadeh A., & Kabiri M.(2017), Applying Bi-factor Multidimensional Item-response Theory Model for Dimensionality and Differential items Functioning Analysis on Testlet-Based Tests. Quarterly of Educational Measurement, 7(28), 121-153.
Alimirzaie M., Moghadam-Zadeh A., Minaei A., Ezanloo B. & Salehi K. (2019). Sources of Question Differential Function and its Application in Education. Journal of Research in Teaching, 7(1), 133-153.
Alshehri, Y. M. (2020). Empirical Evaluation for Gender-Based DIF in an International Assessment: The Case of PIRLS (Doctoral dissertation, University of Kansas).
American Educational Research Association. (2018). Standards for educational and psychological testing. American Educational Research Association.
Anderson, R. C., & Pearson, P. D. (1984). A Schema-Theoretic View of Basic Processes in Reading Comprehension. Technical Report No. 306.
Anderson, R. C., & Pearson, P. D. (1988). A schema-theoretic view of basic processes in reading comprehension. Interactive approaches to second language reading, 37-55.
Asgari, A. (2007). New Approaches to Psychometrics Part Three: Models of Item Response Theory, Rasch Models. Journal of Iranian Psychologists, 4(13), 80-83.
Barro, R. J., & Lee, J. W. (2001). International data on educational attainment: updates and implications. Oxford Economic papers, 53(3), 541-563.
Bauer, D. J. (2017). A more general model for testing measurement invariance and differential item functioning. Psychological methods, 22(3), 507.
Bollmann, S., Berger, M., & Tutz, G. (2017). Item-Focused Trees for the Detection of Differential Item Functioning in Partial Credit Models. Educational and Psychological Measurement, 1-24.
Coulombe, S., Tremblay, J. F., & Marchand, S. (2004). Literacy scores, human capital and growth across fourteen OECD countries (Vol. 89). Geological Survey of Canada.
Desjardins, C. D., & Bulut, O. (2018). Handbook of educational measurement and psychometrics using R. CRC Press.
Finch, W. H., & French, B. F. (2007). Detection of crossing differential item functioning: A comparison of four methods. Educational and psychological Measurement, 67(4), 565-582.
Finch, W. H., Hernández Finch, M. E., & French, B. F. (2016). Recursive partitioning to identify potential causes of differential item functioning in cross-national data. International Journal of Testing, 16(1), 21-53.
Fischer, G. H., & Molenaar, I. W. & (Eds.). (1995). Rasch Models: Foundations، Recent Developments and Applications. New York: Springer-Verlag.
Frick, H., Strobl, C., & Zeileis, A. (2015). Rasch mixture models for DIF detection: A comparison of old and new score specifications. Educational and Psychological Measurement, 75(2), 208-234.
Geske, A., & Ozola, A. (2010). Differential item functioning in the aspect of gender differences in reading literacy. In The 4th IEA International Research Conference.
Giraud, F. (2007). Christian Baudelot, Roger Establet, Quoi de neuf chez les filles? Entre stéréotypes et libertés. Lectures.
Gustafsson, J. E. (1980). Testing and obtaining fit of data to the Rasch model. British Journal of Mathematical and Statistical Psychology, 33(2), 205-233.
Hambleton, R. K., Swaminathan, H., & Rogers, H. J. (1991). Fundamentals of item response theory (Vol. 2). Sage.
Jeffers, H. (2020). Using rasch tree to detect uniform differential item functioning and item difficulty parameters in the progress in international reading literacy data (PIRLS). Ball State University.
Kabiri M., Karimi A., & Bakhshali-Zadeh S. (2017).  PIRLS 2016 findings, Madreseh Publications, Tehran, Iran.
Kamata, A., & Vaughn, B. K. (2004). An Introduction to Differential Item Functioning Analysis. Learning Disabilities: A Contemporary Journal, 2(2), 49-69.
Karami, H., & Salmani Nodoushan, M. A. (2011). Differential Item Functioning (DIF): Current Problems and Future Directions. International Journal of Language Studies (IJLS), 5(3), 133-142.
Kennedy, A. M. (2008). Examining gender and fourth graders’ reading habits and attitudes in PIRLS 2001 and 2006. In 3rd IEA International Research Conference, Taipei, Chinese Taipei.
Millsap, R. E. (2012). Statistical approaches to measurement invariance. Routledge.
Minaie A. (2015). Aapplication of Rasch measurement model to evaluate measurement properties of the Test of Visual- Motor Skills-Revised, Educational Measurement, 5(18), 77-114
Minaei A. & Ghafari Z., (2015) Differential Items functioning of Mathematics Test of TIMSS 2011 in Grade 8 between Girls and Boys using Item-Response Theory Approach (IRT). Educational Measurement, 6(21), 21-40
Mullis, I. V. S., Martin, M. O., Foy, P., & Hooper, M. (2017). PIRLS 2016 International Results in Reading. Retrieved from Boston College, TIMSS & PIRLS International Study Center website: http://timssandpirls.bc.edu/pirls2016/international-results/
Mullis, I. V., Kennedy, A. M., Martin, M. O., & Sainsbury, M. (2006). Assessment Framework and Specifications. PIRLS 2006. International Association for the Evaluation of Educational Achievement. Herengracht 487, Amsterdam, 1017 BT, The Netherlands.
Roux, K. (2020). Examining the equivalence of the PIRLS 2016 released texts in South Africa across three languages (Doctoral dissertation, University of Pretoria).
Sari, I. M. (2020). Decomposing Gender Differences in Reading: Evidence from Nordic PIRLS 2016 (Master's thesis).
Schulz-Heidorf, K., & Støle, H. (2018). Gender differences in Norwegian PIRLS 2016 and ePIRLS 2016 results at test mode, text and item format level. Nordic Journal of Literacy Research, 4(1).
Strobl, C., Kopf, J., & Zeileis, A. (2011a). A new method for detecting differential item functioning in the Rasch model. Innsbruck: University of Innsbruck Working Papers in Economic and Statistic.
Strobl, C., Kopf, J., & Zeileis, A. (2011b) Using the raschtree function for detecting differential item functioning in the Rasch model.‏ Innsbruck: University of Innsbruck Working Papers in Economic and Statistic.
Strobl, C., Kopf, J., & Zeileis, A. (2015). Rasch trees: A new method for detecting differential item functioning in the Rasch model. Psychometrika, 80(2), 289-316.
Stubbe, T. C. (2011). How do different versions of a test instrument function in a single language? A DIF analysis of the PIRLS 2006 German assessments. Educational Research and Evaluation, 17(6), 465-481.
Thissen, D., Steinberg, L., & Wainer, H. (1993). Detection of differential item functioning using the parameters of item response models.‏
Westers, P., & Kelderman, H. (1992). Examining differential item functioning due to item difficulty and alternative attractiveness. Psychometrika, 57(1), 107-118.
Zeileis, A., Strobl, C., Wickelmaier, F., & Kopf, J. (2010). psychotree: Recursive partitioning based on psychometric models. R package version 0.11-1, URL http://CRAN. R-project. org/package= psychotree. University of Innsbruck-Working Papers in Economics and Statistics Recent Papers.
Zumbo, B. D. (1999). A handbook on the theory and methods of differential item functioning (DIF). Ottawa: National Defense Headquarters, 160.