Evaluating AI Competence in Specialized Medicine: Comparative Analysis of ChatGPT and Neurologists in a Neurology Specialist Examination in Spain
JMIR Medical Education · 2024
DOI: 10.2196/56762 ↗Kurz gefasst
Vergleichsstudie in der Zeitschrift JMIR Medical Education, 2024. Verfasst von Pablo Ros-Arlanzón und 1 weiteren Autor:innen. Beteiligte Einrichtung: Department of Neurology, Dr. Balmis General University Hospital, C/ Pintor Baeza, Nº 11, Alicante, 03010, Spain, 34 965933000. Hauptthemen laut MEDLINE: Artificial Intelligence, Neurologists, Neurology, Clinical Competence. Die Arbeit wurde bisher 9-mal zitiert (Crossref) und nennt 31 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
With the rapid advancement of artificial intelligence (AI) in various fields, evaluating its application in specialized medical contexts becomes crucial. ChatGPT, a large language model developed by OpenAI, has shown potential in diverse applications, including medicine.
Objective
This study aims to compare the performance of ChatGPT with that of attending neurologists in a real neurology specialist examination conducted in the Valencian Community, Spain, assessing the AI’s capabilities and limitations in medical knowledge.
Methods
We conducted a comparative analysis using the 2022 neurology specialist examination results from 120 neurologists and responses generated by ChatGPT versions 3.5 and 4. The examination consisted of 80 multiple-choice questions, with a focus on clinical neurology and health legislation. Questions were classified according to Bloom’s Taxonomy. Statistical analysis of performance, including the κ coefficient for response consistency, was performed.
Results
Human participants exhibited a median score of 5.91 (IQR: 4.93-6.76), with 32 neurologists failing to pass. ChatGPT-3.5 ranked 116th out of 122, answering 54.5% of questions correctly (score 3.94). ChatGPT-4 showed marked improvement, ranking 17th with 81.8% of correct answers (score 7.57), surpassing several human specialists. No significant variations were observed in the performance on lower-order questions versus higher-order questions. Additionally, ChatGPT-4 demonstrated increased interrater reliability, as reflected by a higher κ coefficient of 0.73, compared to ChatGPT-3.5’s coefficient of 0.69.
Conclusions
This study underscores the evolving capabilities of AI in medical knowledge assessment, particularly in specialized fields. ChatGPT-4’s performance, outperforming the median score of human participants in a rigorous neurology examination, represents a significant milestone in AI development, suggesting its potential as an effective tool in specialized medical education and assessment.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Neurology, Dr. Balmis General University Hospital, C/ Pintor Baeza, Nº 11, Alicante, 03010, Spain, 34 965933000.
- Department of Neuroscience, Instituto de Investigación Sanitaria y Biomédica de Alicante, Alicante, Spain.
- Department of Clinical Medicine, Miguel Hernández University, Alicante, Spain.
ORCID-Profile: Pablo Ros-Arlanzón, Angel Perez-Sempere
Zitieren
APA
Ros-Arlanzón, P., & Perez-Sempere, A. (2024). Evaluating AI Competence in Specialized Medicine: Comparative Analysis of ChatGPT and Neurologists in a Neurology Specialist Examination in Spain. JMIR Medical Education. https://doi.org/10.2196/56762
Vancouver
Ros-Arlanzón P, Perez-Sempere A. Evaluating AI Competence in Specialized Medicine: Comparative Analysis of ChatGPT and Neurologists in a Neurology Specialist Examination in Spain. JMIR Medical Education. 2024. doi:10.2196/56762
BibTeX
@article{RosArlanzn2024,
author = {Ros-Arlanzón, Pablo and Perez-Sempere, Angel},
title = {Evaluating AI Competence in Specialized Medicine: Comparative Analysis of ChatGPT and Neurologists in a Neurology Specialist Examination in Spain},
journal = {JMIR Medical Education},
year = {2024},
doi = {10.2196/56762}
}
EUVIMED