Performance of ChatGPT on Nursing Licensure Examinations in the United States and China: Cross-Sectional Study
JMIR Medical Education · 03.10.2024 · 10 · e52746-e52746
DOI: 10.2196/52746 ↗Kurz gefasst
Querschnittstudie in der Zeitschrift JMIR Medical Education, online erschienen am 03.10.2024. Verfasst von Zelin Wu und 5 weiteren Autor:innen. Beteiligte Einrichtung: Department of Bone and Joint Surgery and Sports Medicine Center, The First Affiliated Hospital, Guangzhou, China. Hauptthemen laut MEDLINE: Licensure, Nursing, Educational Measurement. Die Arbeit wurde bisher 36-mal zitiert (Crossref) und nennt 53 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
The creation of large language models (LLMs) such as ChatGPT is an important step in the development of artificial intelligence, which shows great potential in medical education due to its powerful language understanding and generative capabilities. The purpose of this study was to quantitatively evaluate and comprehensively analyze ChatGPT’s performance in handling questions for the National Nursing Licensure Examination (NNLE) in China and the United States, including the National Council Licensure Examination for Registered Nurses (NCLEX-RN) and the NNLE.
Objective
This study aims to examine how well LLMs respond to the NCLEX-RN and the NNLE multiple-choice questions (MCQs) in various language inputs. To evaluate whether LLMs can be used as multilingual learning assistance for nursing, and to assess whether they possess a repository of professional knowledge applicable to clinical nursing practice.
Methods
First, we compiled 150 NCLEX-RN Practical MCQs, 240 NNLE Theoretical MCQs, and 240 NNLE Practical MCQs. Then, the translation function of ChatGPT 3.5 was used to translate NCLEX-RN questions from English to Chinese and NNLE questions from Chinese to English. Finally, the original version and the translated version of the MCQs were inputted into ChatGPT 4.0, ChatGPT 3.5, and Google Bard. Different LLMs were compared according to the accuracy rate, and the differences between different language inputs were compared.
Results
The accuracy rates of ChatGPT 4.0 for NCLEX-RN practical questions and Chinese-translated NCLEX-RN practical questions were 88.7% (133/150) and 79.3% (119/150), respectively. Despite the statistical significance of the difference ( P =.03), the correct rate was generally satisfactory. Around 71.9% (169/235) of NNLE Theoretical MCQs and 69.1% (161/233) of NNLE Practical MCQs were correctly answered by ChatGPT 4.0. The accuracy of ChatGPT 4.0 in processing NNLE Theoretical MCQs and NNLE Practical MCQs translated into English was 71.5% (168/235; P =.92) and 67.8% (158/233; P =.77), respectively, and there was no statistically significant difference between the results of text input in different languages. ChatGPT 3.5 (NCLEX-RN P =.003, NNLE Theoretical P
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Bone and Joint Surgery and Sports Medicine Center, The First Affiliated Hospital, Guangzhou, China.
- Department of Joint Surgery and Sports Medicine, Zhuhai People's Hospital, Zhuhai City, China.
- School of Nursing, Yangzhou University, Yangzhou, China.
ORCID-Profile: Zelin Wu, Wenyi Gan, Zhaowen Xue, Zhengxin Ni, Xiaofei Zheng, Yiyi Zhang
Zitieren
APA
Wu, Z., Gan, W., Xue, Z., Ni, Z., Zheng, X., & Zhang, Y. (2024). Performance of ChatGPT on Nursing Licensure Examinations in the United States and China: Cross-Sectional Study. JMIR Medical Education, 10, e52746-e52746. https://doi.org/10.2196/52746
Vancouver
Wu Z, Gan W, Xue Z, Ni Z, Zheng X, Zhang Y. Performance of ChatGPT on Nursing Licensure Examinations in the United States and China: Cross-Sectional Study. JMIR Medical Education. 2024;10:e52746-e52746. doi:10.2196/52746
BibTeX
@article{Wu2024,
author = {Wu, Zelin and Gan, Wenyi and Xue, Zhaowen and Ni, Zhengxin and Zheng, Xiaofei and Zhang, Yiyi},
title = {Performance of ChatGPT on Nursing Licensure Examinations in the United States and China: Cross-Sectional Study},
journal = {JMIR Medical Education},
year = {2024},
volume = {10},
pages = {e52746--e52746},
doi = {10.2196/52746}
}
EUVIMED