EUVIMEDEUVIMED entdecken
QuerschnittstudieOpen Access

Performance of ChatGPT on Nursing Licensure Examinations in the United States and China: Cross-Sectional Study

Zelin Wu iD, Wenyi Gan iD, Zhaowen Xue iD, Zhengxin Ni iD, Xiaofei Zheng iD, Yiyi Zhang iD

JMIR Medical Education · 03.10.2024 · 10 · e52746-e52746

DOI: 10.2196/52746 ↗

Kurz gefasst

Querschnittstudie in der Zeitschrift JMIR Medical Education, online erschienen am 03.10.2024. Verfasst von Zelin Wu und 5 weiteren Autor:innen. Beteiligte Einrichtung: Department of Bone and Joint Surgery and Sports Medicine Center, The First Affiliated Hospital, Guangzhou, China. Hauptthemen laut MEDLINE: Licensure, Nursing, Educational Measurement. Die Arbeit wurde bisher 36-mal zitiert (Crossref) und nennt 53 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

The creation of large language models (LLMs) such as ChatGPT is an important step in the development of artificial intelligence, which shows great potential in medical education due to its powerful language understanding and generative capabilities. The purpose of this study was to quantitatively evaluate and comprehensively analyze ChatGPT’s performance in handling questions for the National Nursing Licensure Examination (NNLE) in China and the United States, including the National Council Licensure Examination for Registered Nurses (NCLEX-RN) and the NNLE.

Objective

This study aims to examine how well LLMs respond to the NCLEX-RN and the NNLE multiple-choice questions (MCQs) in various language inputs. To evaluate whether LLMs can be used as multilingual learning assistance for nursing, and to assess whether they possess a repository of professional knowledge applicable to clinical nursing practice.

Methods

First, we compiled 150 NCLEX-RN Practical MCQs, 240 NNLE Theoretical MCQs, and 240 NNLE Practical MCQs. Then, the translation function of ChatGPT 3.5 was used to translate NCLEX-RN questions from English to Chinese and NNLE questions from Chinese to English. Finally, the original version and the translated version of the MCQs were inputted into ChatGPT 4.0, ChatGPT 3.5, and Google Bard. Different LLMs were compared according to the accuracy rate, and the differences between different language inputs were compared.

Results

The accuracy rates of ChatGPT 4.0 for NCLEX-RN practical questions and Chinese-translated NCLEX-RN practical questions were 88.7% (133/150) and 79.3% (119/150), respectively. Despite the statistical significance of the difference ( P =.03), the correct rate was generally satisfactory. Around 71.9% (169/235) of NNLE Theoretical MCQs and 69.1% (161/233) of NNLE Practical MCQs were correctly answered by ChatGPT 4.0. The accuracy of ChatGPT 4.0 in processing NNLE Theoretical MCQs and NNLE Practical MCQs translated into English was 71.5% (168/235; P =.92) and 67.8% (158/233; P =.77), respectively, and there was no statistically significant difference between the results of text input in different languages. ChatGPT 3.5 (NCLEX-RN P =.003, NNLE Theoretical P

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Licensure, NursingEducational MeasurementChinaHumansCross-Sectional StudiesUnited StatesArtificial Intelligence

Stichwörter der Autor:innen

AIChatGPTChinaLLMsUSAUnited States of Americaaccuracy rateartificial intelligenceauxiliary toollarge language modellarge language modelslicensingnursingnursing educationnursing licensure examination

Autor:innen und Institutionen

  • Department of Bone and Joint Surgery and Sports Medicine Center, The First Affiliated Hospital, Guangzhou, China.
  • Department of Joint Surgery and Sports Medicine, Zhuhai People's Hospital, Zhuhai City, China.
  • School of Nursing, Yangzhou University, Yangzhou, China.

ORCID-Profile: Zelin Wu, Wenyi Gan, Zhaowen Xue, Zhengxin Ni, Xiaofei Zheng, Yiyi Zhang

Zitieren

APA

Wu, Z., Gan, W., Xue, Z., Ni, Z., Zheng, X., & Zhang, Y. (2024). Performance of ChatGPT on Nursing Licensure Examinations in the United States and China: Cross-Sectional Study. JMIR Medical Education, 10, e52746-e52746. https://doi.org/10.2196/52746

Vancouver

Wu Z, Gan W, Xue Z, Ni Z, Zheng X, Zhang Y. Performance of ChatGPT on Nursing Licensure Examinations in the United States and China: Cross-Sectional Study. JMIR Medical Education. 2024;10:e52746-e52746. doi:10.2196/52746

BibTeX
@article{Wu2024,
  author = {Wu, Zelin and Gan, Wenyi and Xue, Zhaowen and Ni, Zhengxin and Zheng, Xiaofei and Zhang, Yiyi},
  title = {Performance of ChatGPT on Nursing Licensure Examinations in the United States and China: Cross-Sectional Study},
  journal = {JMIR Medical Education},
  year = {2024},
  volume = {10},
  pages = {e52746--e52746},
  doi = {10.2196/52746}
}
Kostenlos starten →