Psychometric Analysis of a Complex Multiple-Choice Instrument for Measuring Critical Thinking Skills on Sound Waves Using Item Response Theory

Abstract

Critical thinking skills are an essential 21st-century competence, yet valid and reliable measurement instruments in physics remain limited in Indonesia. This descriptive quantitative study aims to analyze the psychometric characteristics of a complex multiple-choice instrument designed to measure students' critical thinking skills in sound waves. The study involved 187 eleventh-grade students from four senior high schools in West Java. The instrument consisted of 20 complex multiple-choice items developed based on Ennis’s critical thinking framework (covering five aspects of critical thinking) and four subtopics of sound waves. Content validity was evaluated using Aiken’s V index, while empirical validity was analyzed using the Rasch model. Item Response Theory (IRT) was applied by comparing the 1PL, 2PL, and 3PL models. The results showed that the instrument met the unidimensionality assumption with high content validity (Aiken’s V = 0.939). Based on the Rasch model fit criteria, 19 out of 20 items (95%) were empirically valid. Among the IRT models, the 2PL model provided the best fit (AIC = 3637.95). The item difficulty parameters ranged from -3.878 to -0.153 logits, and the discrimination parameters ranged from 0.480 to 1.613. The test yielded a marginal reliability of 0.6946 and achieved maximum information at an ability level of θ = -1.55 logits, indicating that this instrument accurately measures students with low to moderate abilities. Overall, the instrument demonstrates satisfactory psychometric characteristics and stands as one of the first comprehensively developed tools to measure critical thinking in sound waves using IRT model comparison. Future studies are suggested to add items with higher difficulty levels to improve measurement accuracy for high-ability students.