Performance of ChatGPT on Nursing Licensure Examinations in the United States and China: Cross-Sectional Study
JMIR Medical Education · 2024-10-03 · 10 · e52746-e52746
DOI: 10.2196/52746 ↗At a glance
Querschnittstudie in JMIR Medical Education, published online on 2024-10-03. Written by Zelin Wu and 5 co-authors. Institution: Department of Bone and Joint Surgery and Sports Medicine Center, The First Affiliated Hospital, Guangzhou, China. Major MEDLINE topics: Licensure, Nursing, Educational Measurement. Cited 36 times so far (Crossref); 53 references. The full text is freely available in PubMed Central.
Abstract (original language)
Background
The creation of large language models (LLMs) such as ChatGPT is an important step in the development of artificial intelligence, which shows great potential in medical education due to its powerful language understanding and generative capabilities. The purpose of this study was to quantitatively evaluate and comprehensively analyze ChatGPT’s performance in handling questions for the National Nursing Licensure Examination (NNLE) in China and the United States, including the National Council Licensure Examination for Registered Nurses (NCLEX-RN) and the NNLE.
Objective
This study aims to examine how well LLMs respond to the NCLEX-RN and the NNLE multiple-choice questions (MCQs) in various language inputs. To evaluate whether LLMs can be used as multilingual learning assistance for nursing, and to assess whether they possess a repository of professional knowledge applicable to clinical nursing practice.
Methods
First, we compiled 150 NCLEX-RN Practical MCQs, 240 NNLE Theoretical MCQs, and 240 NNLE Practical MCQs. Then, the translation function of ChatGPT 3.5 was used to translate NCLEX-RN questions from English to Chinese and NNLE questions from Chinese to English. Finally, the original version and the translated version of the MCQs were inputted into ChatGPT 4.0, ChatGPT 3.5, and Google Bard. Different LLMs were compared according to the accuracy rate, and the differences between different language inputs were compared.
Results
The accuracy rates of ChatGPT 4.0 for NCLEX-RN practical questions and Chinese-translated NCLEX-RN practical questions were 88.7% (133/150) and 79.3% (119/150), respectively. Despite the statistical significance of the difference ( P =.03), the correct rate was generally satisfactory. Around 71.9% (169/235) of NNLE Theoretical MCQs and 69.1% (161/233) of NNLE Practical MCQs were correctly answered by ChatGPT 4.0. The accuracy of ChatGPT 4.0 in processing NNLE Theoretical MCQs and NNLE Practical MCQs translated into English was 71.5% (168/235; P =.92) and 67.8% (158/233; P =.77), respectively, and there was no statistically significant difference between the results of text input in different languages. ChatGPT 3.5 (NCLEX-RN P =.003, NNLE Theoretical P
Subjects
MeSH terms (MEDLINE)
Author keywords
Authors and institutions
- Department of Bone and Joint Surgery and Sports Medicine Center, The First Affiliated Hospital, Guangzhou, China.
- Department of Joint Surgery and Sports Medicine, Zhuhai People's Hospital, Zhuhai City, China.
- School of Nursing, Yangzhou University, Yangzhou, China.
ORCID profiles: Zelin Wu, Wenyi Gan, Zhaowen Xue, Zhengxin Ni, Xiaofei Zheng, Yiyi Zhang
Cite
APA
Wu, Z., Gan, W., Xue, Z., Ni, Z., Zheng, X., & Zhang, Y. (2024). Performance of ChatGPT on Nursing Licensure Examinations in the United States and China: Cross-Sectional Study. JMIR Medical Education, 10, e52746-e52746. https://doi.org/10.2196/52746
Vancouver
Wu Z, Gan W, Xue Z, Ni Z, Zheng X, Zhang Y. Performance of ChatGPT on Nursing Licensure Examinations in the United States and China: Cross-Sectional Study. JMIR Medical Education. 2024;10:e52746-e52746. doi:10.2196/52746
BibTeX
@article{Wu2024,
author = {Wu, Zelin and Gan, Wenyi and Xue, Zhaowen and Ni, Zhengxin and Zheng, Xiaofei and Zhang, Yiyi},
title = {Performance of ChatGPT on Nursing Licensure Examinations in the United States and China: Cross-Sectional Study},
journal = {JMIR Medical Education},
year = {2024},
volume = {10},
pages = {e52746--e52746},
doi = {10.2196/52746}
}
EUVIMED