EUVIMEDDescoperă EUVIMED
QuerschnittstudieOpen access

Performance of ChatGPT-4 on Taiwanese Traditional Chinese Medicine Licensing Examinations: Cross-Sectional Study

Liang-Wei Tseng iD, Yi-Chin Lu iD, Liang-Chi Tseng iD, Yu-Chun Chen iD, Hsing-Yu Chen iD

JMIR Medical Education · 2025-03-19 · 11 · e58897-e58897

DOI: 10.2196/58897 ↗

At a glance

Querschnittstudie in JMIR Medical Education, published online on 2025-03-19. Written by Liang-Wei Tseng and 4 co-authors. Institution: Division of Chinese Acupuncture and Traumatology, Center of Traditional Chinese Medicine, Chang Gung Memorial Hospital, Taoyuan, Taiwan. Major MEDLINE topics: Medicine, Chinese Traditional, Educational Measurement, Artificial Intelligence, Licensure, Medical. Cited 15 times so far (Crossref); 48 references. The full text is freely available in PubMed Central.

RezumatSubjectsDate bibliograficeCiteRelated studies

Abstract (original language)

Background

The integration of artificial intelligence (AI), notably ChatGPT, into medical education, has shown promising results in various medical fields. Nevertheless, its efficacy in traditional Chinese medicine (TCM) examinations remains understudied.

Objective

This study aims to (1) assess the performance of ChatGPT on the TCM licensing examination in Taiwan and (2) evaluate the model’s explainability in answering TCM-related questions to determine its suitability as a TCM learning tool.

Methods

We used the GPT-4 model to respond to 480 questions from the 2022 TCM licensing examination. This study compared the performance of the model against that of licensed TCM doctors using 2 approaches, namely direct answer selection and provision of explanations before answer selection. The accuracy and consistency of AI-generated responses were analyzed. Moreover, a breakdown of question characteristics was performed based on the cognitive level, depth of knowledge, types of questions, vignette style, and polarity of questions.

Results

ChatGPT achieved an overall accuracy of 43.9%, which was lower than that of 2 human participants (70% and 78.4%). The analysis did not reveal a significant correlation between the accuracy of the model and the characteristics of the questions. An in-depth examination indicated that errors predominantly resulted from a misunderstanding of TCM concepts (55.3%), emphasizing the limitations of the model with regard to its TCM knowledge base and reasoning capability.

Conclusions

Although ChatGPT shows promise as an educational tool, its current performance on TCM licensing examinations is lacking. This highlights the need for enhancing AI models with specialized TCM training and suggests a cautious approach to utilizing AI for TCM education. Future research should focus on model improvement and the development of tailored educational applications to support TCM learning.

Subjects

MeSH terms (MEDLINE)

Medicine, Chinese TraditionalEducational MeasurementArtificial IntelligenceLicensure, MedicalHumansTaiwanCross-Sectional StudiesFemaleMaleAdultGenerative Artificial Intelligence

Author keywords

AI language understanding toolsChatGPTChinese medical licensing examinationChinese medicine license examartificial intelligencelarge language modelmachine learningmedical educationnatural language processingtraditional Chinese medicine

Authors and institutions

  • Division of Chinese Acupuncture and Traumatology, Center of Traditional Chinese Medicine, Chang Gung Memorial Hospital, Taoyuan, Taiwan.
  • Division of Chinese Internal Medicine, Center for Traditional Chinese Medicine, Chang Gung Memorial Hospital, No. 123, Dinghu Rd, Gueishan Dist, Taoyuan, 33378, Taiwan, 886 3 3196200 ext 2611, 886 3 3298995.
  • Google International LLC Taiwan Branch, Taipei, Taiwan.
  • School of Medicine, Faculty of Medicine, National Yang-Ming Chiao Tung University, Taipei, Taiwan.
  • Taipei Veterans General Hospital, Yuli Branch, Taipei, Taiwan.
  • Institute of Hospital and Health Care Administration, National Yang-Ming Chiao Tung University, Taipei, Taiwan.

ORCID profiles: Liang-Wei Tseng, Yi-Chin Lu, Liang-Chi Tseng, Yu-Chun Chen, Hsing-Yu Chen

Cite

APA

Tseng, L. W., Lu, Y. C., Tseng, L. C., Chen, Y. C., & Chen, H. Y. (2025). Performance of ChatGPT-4 on Taiwanese Traditional Chinese Medicine Licensing Examinations: Cross-Sectional Study. JMIR Medical Education, 11, e58897-e58897. https://doi.org/10.2196/58897

Vancouver

Tseng LW, Lu YC, Tseng LC, Chen YC, Chen HY. Performance of ChatGPT-4 on Taiwanese Traditional Chinese Medicine Licensing Examinations: Cross-Sectional Study. JMIR Medical Education. 2025;11:e58897-e58897. doi:10.2196/58897

BibTeX
@article{Tseng2025,
  author = {Tseng, Liang-Wei and Lu, Yi-Chin and Tseng, Liang-Chi and Chen, Yu-Chun and Chen, Hsing-Yu},
  title = {Performance of ChatGPT-4 on Taiwanese Traditional Chinese Medicine Licensing Examinations: Cross-Sectional Study},
  journal = {JMIR Medical Education},
  year = {2025},
  volume = {11},
  pages = {e58897--e58897},
  doi = {10.2196/58897}
}
Start for free →