Comparison of real data and simulated data analysis of a stopping rule based on the standard error of measurement in computerized adaptive testing for medical examinations in Korea: a psychometric study
Journal of Educational Evaluation for Health Professions · 2024
DOI: 10.3352/jeehp.2024.21.18 ↗Kurz gefasst
Vergleichsstudie in der Zeitschrift Journal of Educational Evaluation for Health Professions, 2024. Verfasst von Dong Gi Seo und 2 weiteren Autor:innen. Beteiligte Einrichtung: Department of Psychology, Hallym Applied Psychology Institute, College of Social Science, Hallym University, Chuncheon, Korea. Hauptthemen laut MEDLINE: Educational Measurement, Psychometrics, Students, Medical. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 10 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Hallym University.
Abstract (Originalsprache)
Purpose
This study aimed to compare and evaluate the efficiency and accuracy of computerized adaptive testing (CAT) under 2 stopping rules (standard error of measurement [SEM]=0.3 and 0.25) using both real and simulated data in medical examinations in Korea.
Methods
This study employed post-hoc simulation and real data analysis to explore the optimal stopping rule for CAT in medical examinations. The real data were obtained from the responses of 3rd-year medical students during examinations in 2020 at Hallym University College of Medicine. Simulated data were generated using estimated parameters from a real item bank in R. Outcome variables included the number of examinees’ passing or failing with SEM values of 0.25 and 0.30, the number of items administered, and the correlation. The consistency of real CAT result was evaluated by examining consistency of pass or fail based on a cut score of 0.0. The efficiency of all CAT designs was assessed by comparing the average number of items administered under both stopping rules.
Results
Both SEM 0.25 and SEM 0.30 provided a good balance between accuracy and efficiency in CAT. The real data showed minimal differences in pass/fail outcomes between the 2 SEM conditions, with a high correlation (r=0.99) between ability estimates. The simulation results confirmed these findings, indicating similar average item numbers between real and simulated data.
Conclusion
The findings suggest that both SEM 0.25 and 0.30 are effective termination criteria in the context of the Rasch model, balancing accuracy and efficiency in CAT.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Psychology, Hallym Applied Psychology Institute, College of Social Science, Hallym University, Chuncheon, Korea.
- The CAT Korea Company, Chuncheon, Korea.
Förderung
- Hallym University
Zitieren
APA
Seo, D. G., Choi, J., & Kim, J. (2024). Comparison of real data and simulated data analysis of a stopping rule based on the standard error of measurement in computerized adaptive testing for medical examinations in Korea: a psychometric study. Journal of Educational Evaluation for Health Professions. https://doi.org/10.3352/jeehp.2024.21.18
Vancouver
Seo DG, Choi J, Kim J. Comparison of real data and simulated data analysis of a stopping rule based on the standard error of measurement in computerized adaptive testing for medical examinations in Korea: a psychometric study. Journal of Educational Evaluation for Health Professions. 2024. doi:10.3352/jeehp.2024.21.18
BibTeX
@article{Seo2024,
author = {Seo, Dong Gi and Choi, Jeongwook and Kim, Jinha},
title = {Comparison of real data and simulated data analysis of a stopping rule based on the standard error of measurement in computerized adaptive testing for medical examinations in Korea: a psychometric study},
journal = {Journal of Educational Evaluation for Health Professions},
year = {2024},
doi = {10.3352/jeehp.2024.21.18}
}
EUVIMED