EUVIMEDEUVIMED entdecken
VergleichsstudieOpen Access · CC BY 4.0

Comparing generative artificial intelligence platforms and nursing student performance on a women’s health nursing examination in Korea: a Rasch model approach

Eun Jeong Ko, Tae Kyung Lee, Geum Hee Jeong

Journal of Educational Evaluation for Health Professions · 2025

DOI: 10.3352/jeehp.2025.22.23 ↗

Kurz gefasst

Vergleichsstudie in der Zeitschrift Journal of Educational Evaluation for Health Professions, 2025. Verfasst von Eun Jeong Ko und 2 weiteren Autor:innen. Beteiligte Einrichtung: School of Nursing and Research Institute in Nursing Science, Hallym University, Chuncheon, Korea. Hauptthemen laut MEDLINE: Students, Nursing, Artificial Intelligence, Educational Measurement. Die Arbeit wurde bisher 2-mal zitiert (Crossref). Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Hallym University.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Purpose

This psychometric study aimed to compare the ability parameter estimates of generative artificial intelligence (AI) platforms with those of nursing students on a 50-item women’s health nursing examination at Hallym University, Korea, using the Rasch model. It also sought to estimate item difficulty parameters and evaluate AI performance across varying difficulty levels.

Methods

The exam, consisting of 39 multiple-choice items and 11 true/false items, was administered to 111 fourth-year nursing students in June 2023. In December 2024, 6 generative AI platforms (GPT-4o, ChatGPT Free, Claude.ai, Clova X, Mistral.ai, Google Gemini) completed the same items. The responses were analyzed using the Rasch model to estimate the ability and difficulty parameters. Unidimensionality was verified by the Dimensionality Evaluation to Enumerate Contributing Traits (DETECT), and analyses were conducted using the R packages irtQ and TAM.

Results

The items satisfied unidimensionality (DETECT=–0.16). Item difficulty parameter estimates ranged from –3.87 to 1.96 logits (mean=–0.61), with a mean difficulty index of 0.79. Examinees’ ability parameter estimates ranged from –0.71 to 3.14 logits (mean=1.17). GPT-4o, ChatGPT Free, and Claude.ai outperformed the median student ability (1.09 logits), scoring 2.68, 2.34, and 2.34, respectively, while Clova X, Mistral.ai, and Google Gemini exhibited lower scores (0.20, –0.12, 0.80). The test information curve peaked below θ=0, indicating suitability for examinees with low to average ability.

Conclusion

Advanced generative AI platforms approximated the performance of high-performing students, but outcomes varied. The Rasch model effectively evaluated AI competency, supporting its potential utility for future AI performance assessments in nursing education.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Students, NursingArtificial IntelligenceEducational MeasurementHumansRepublic of KoreaFemalePsychometricsEducation, Nursing, BaccalaureateAdultGenerative Artificial Intelligence

Stichwörter der Autor:innen

Artificial intelligenceNursing educationNursing studentsPsychometricsRepublic of Korea

Autor:innen und Institutionen

  • School of Nursing and Research Institute in Nursing Science, Hallym University, Chuncheon, Korea.
  • Kangwon National University Hospital, Chuncheon, Korea.

Förderung

  • Hallym University

Zitieren

APA

Ko, E. J., Lee, T. K., & Jeong, G. H. (2025). Comparing generative artificial intelligence platforms and nursing student performance on a women’s health nursing examination in Korea: a Rasch model approach. Journal of Educational Evaluation for Health Professions. https://doi.org/10.3352/jeehp.2025.22.23

Vancouver

Ko EJ, Lee TK, Jeong GH. Comparing generative artificial intelligence platforms and nursing student performance on a women’s health nursing examination in Korea: a Rasch model approach. Journal of Educational Evaluation for Health Professions. 2025. doi:10.3352/jeehp.2025.22.23

BibTeX
@article{Ko2025,
  author = {Ko, Eun Jeong and Lee, Tae Kyung and Jeong, Geum Hee},
  title = {Comparing generative artificial intelligence platforms and nursing student performance on a women’s health nursing examination in Korea: a Rasch model approach},
  journal = {Journal of Educational Evaluation for Health Professions},
  year = {2025},
  doi = {10.3352/jeehp.2025.22.23}
}
Kostenlos starten →