Large Language Models in Worldwide Medical Exams: Platform Development and Comprehensive Analysis
Journal of Medical Internet Research · 27.12.2024 · 26 · e66114
DOI: 10.2196/66114 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Medical Internet Research, online erschienen am 27.12.2024. Verfasst von Hui Zong und 9 weiteren Autor:innen. Beteiligte Einrichtung: Joint Laboratory of Artificial Intelligence for Critical Care Medicine, Department of Critical Care Medicine and Institutes for Systems Genetics, Fro…. Hauptthemen laut MEDLINE: Education, Medical. Die Arbeit wurde bisher 92-mal zitiert (Crossref) und nennt 42 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
Large language models (LLMs) are increasingly integrated into medical education, with transformative potential for learning and assessment. However, their performance across diverse medical exams globally has remained underexplored.
Objective
This study aims to introduce MedExamLLM, a comprehensive platform designed to systematically evaluate the performance of LLMs on medical exams worldwide. Specifically, the platform seeks to (1) compile and curate performance data for diverse LLMs on worldwide medical exams; (2) analyze trends and disparities in LLM capabilities across geographic regions, languages, and contexts; and (3) provide a resource for researchers, educators, and developers to explore and advance the integration of artificial intelligence in medical education.
Methods
A systematic search was conducted on April 25, 2024, in the PubMed database to identify relevant publications. Inclusion criteria encompassed peer-reviewed, English-language, original research articles that evaluated at least one LLM on medical exams. Exclusion criteria included review articles, non-English publications, preprints, and studies without relevant data on LLM performance. The screening process for candidate publications was independently conducted by 2 researchers to ensure accuracy and reliability. Data, including exam information, data process information, model performance, data availability, and references, were manually curated, standardized, and organized. These curated data were integrated into the MedExamLLM platform, enabling its functionality to visualize and analyze LLM performance across geographic, linguistic, and exam characteristics. The web platform was developed with a focus on accessibility, interactivity, and scalability to support continuous data updates and user engagement.
Results
A total of 193 articles were included for final analysis. MedExamLLM comprised information for 16 LLMs on 198 medical exams conducted in 28 countries across 15 languages from the year 2009 to the year 2023. The United States accounted for the highest number of medical exams and related publications, with English being the dominant language used in these exams. The Generative Pretrained Transformer (GPT) series models, especially GPT-4, demonstrated superior performance, achieving pass rates significantly higher than other LLMs. The analysis revealed significant variability in the capabilities of LLMs across different geographic and linguistic contexts.
Conclusions
MedExamLLM is an open-source, freely accessible, and publicly available online platform providing comprehensive performance evaluation information and evidence knowledge about LLMs on medical exams around the world. The MedExamLLM platform serves as a valuable resource for educators, researchers, and developers in the fields of clinical medicine and artificial intelligence. By synthesizing evidence on LLM capabilities, the platform provides valuable insights to support the integration of artificial intelligence into medical education. Limitations include potential biases in the data source and the exclusion of non-English literature. Future research should address these gaps and explore methods to enhance LLM performance in diverse contexts.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Joint Laboratory of Artificial Intelligence for Critical Care Medicine, Department of Critical Care Medicine and Institutes for Systems Genetics, Frontiers Science Center for Disease-related Molecular Network, West Chin…
- Shanghai Key Laboratory of Signaling and Disease Research, School of Life Sciences and Technology, Tongji University, Shanghai, China.
- Department of Neurosurgery, First Affiliated Hospital of Xinjiang Medical University, Urumqi, China.
- Department of Urology, West China Hospital, Sichuan University, Chengdu, China.
- West China Tianfu Hospital, Sichuan University, Chengdu, China.
ORCID-Profile: Hui Zong, Rongrong Wu, Jiaxue Cha, Jiao Wang, Erman Wu, Jiakun Li, Yi Zhou, Chi Zhang, Weizhe Feng, Bairong Shen
Zitieren
APA
Zong, H., Wu, R., Cha, J., Wang, J., Wu, E., Li, J., Zhou, Y., Zhang, C., Feng, W., & Shen, B. (2024). Large Language Models in Worldwide Medical Exams: Platform Development and Comprehensive Analysis. Journal of Medical Internet Research, 26, e66114. https://doi.org/10.2196/66114
Vancouver
Zong H, Wu R, Cha J, Wang J, Wu E, Li J, et al. Large Language Models in Worldwide Medical Exams: Platform Development and Comprehensive Analysis. Journal of Medical Internet Research. 2024;26:e66114. doi:10.2196/66114
BibTeX
@article{Zong2024,
author = {Zong, Hui and Wu, Rongrong and Cha, Jiaxue and Wang, Jiao and Wu, Erman and Li, Jiakun and Zhou, Yi and Zhang, Chi and Feng, Weizhe and Shen, Bairong},
title = {Large Language Models in Worldwide Medical Exams: Platform Development and Comprehensive Analysis},
journal = {Journal of Medical Internet Research},
year = {2024},
volume = {26},
pages = {e66114},
doi = {10.2196/66114}
}
EUVIMED