EUVIMEDScopri EUVIMED
Open access

Performance of the Large Language Model ChatGPT on the National Nurse Examinations in Japan: Evaluation Study

Kazuya Taira iD, Takahiro Itaya iD, Ayame Hanada iD

JMIR Nursing · 2023-06-27 · 6 · e47305

DOI: 10.2196/47305 ↗

At a glance

Journal article in JMIR Nursing, published online on 2023-06-27. Written by Kazuya Taira and 2 co-authors. Institution: Department of Human Health Sciences, Graduate School of Medicine, Kyoto University, Kyoto, Japan. Cited 64 times so far (Crossref); 15 references. The full text is freely available in PubMed Central.

AbstractSubjectsDati bibliograficiCiteRelated studies

Abstract (original language)

Background

ChatGPT, a large language model, has shown good performance on physician certification examinations and medical consultations. However, its performance has not been examined in languages other than English or on nursing examinations.

Objective

We aimed to evaluate the performance of ChatGPT on the Japanese National Nurse Examinations.

Methods

We evaluated the percentages of correct answers provided by ChatGPT (GPT-3.5) for all questions on the Japanese National Nurse Examinations from 2019 to 2023, excluding inappropriate questions and those containing images. Inappropriate questions were pointed out by a third-party organization and announced by the government to be excluded from scoring. Specifically, these include “questions with inappropriate question difficulty” and “questions with errors in the questions or choices.” These examinations consist of 240 questions each year, divided into basic knowledge questions that test the basic issues of particular importance to nurses and general questions that test a wide range of specialized knowledge. Furthermore, the questions had 2 types of formats: simple-choice and situation-setup questions. Simple-choice questions are primarily knowledge-based and multiple-choice, whereas situation-setup questions entail the candidate reading a patient’s and family situation’s description, and selecting the nurse's action or patient's response. Hence, the questions were standardized using 2 types of prompts before requesting answers from ChatGPT. Chi-square tests were conducted to compare the percentage of correct answers for each year's examination format and specialty area related to the question. In addition, a Cochran-Armitage trend test was performed with the percentage of correct answers from 2019 to 2023.

Results

The 5-year average percentage of correct answers for ChatGPT was 75.1% (SD 3%) for basic knowledge questions and 64.5% (SD 5%) for general questions. The highest percentage of correct answers on the 2019 examination was 80% for basic knowledge questions and 71.2% for general questions. ChatGPT met the passing criteria for the 2019 Japanese National Nurse Examination and was close to passing the 2020-2023 examinations, with only a few more correct answers required to pass. ChatGPT had a lower percentage of correct answers in some areas, such as pharmacology, social welfare, related law and regulations, endocrinology/metabolism, and dermatology, and a higher percentage of correct answers in the areas of nutrition, pathology, hematology, ophthalmology, otolaryngology, dentistry and dental surgery, and nursing integration and practice.

Conclusions

ChatGPT only passed the 2019 Japanese National Nursing Examination during the most recent 5 years. Although it did not pass the examinations from other years, it performed very close to the passing level, even in those containing questions related to psychology, communication, and nursing.

Subjects

Author keywords

ChatGPTJapanNational Nurse Examinationartificial intelligencenatural language processingregistered nurses

Authors and institutions

  • Department of Human Health Sciences, Graduate School of Medicine, Kyoto University, Kyoto, Japan.
  • Department of Healthcare Epidemiology, Graduate School of Medicine and Public Health, Kyoto University, Kyoto, Japan.
  • Department of Preventive Medicine and Public Health, School of Medicine, Keio University, Tokyo, Japan.

ORCID profiles: Kazuya Taira, Takahiro Itaya, Ayame Hanada

Cite

APA

Taira, K., Itaya, T., & Hanada, A. (2023). Performance of the Large Language Model ChatGPT on the National Nurse Examinations in Japan: Evaluation Study. JMIR Nursing, 6, e47305. https://doi.org/10.2196/47305

Vancouver

Taira K, Itaya T, Hanada A. Performance of the Large Language Model ChatGPT on the National Nurse Examinations in Japan: Evaluation Study. JMIR Nursing. 2023;6:e47305. doi:10.2196/47305

BibTeX
@article{Taira2023,
  author = {Taira, Kazuya and Itaya, Takahiro and Hanada, Ayame},
  title = {Performance of the Large Language Model ChatGPT on the National Nurse Examinations in Japan: Evaluation Study},
  journal = {JMIR Nursing},
  year = {2023},
  volume = {6},
  pages = {e47305},
  doi = {10.2196/47305}
}
Start for free →