EUVIMEDEUVIMED entdecken
VergleichsstudieOpen Access

Evaluating AI Competence in Specialized Medicine: Comparative Analysis of ChatGPT and Neurologists in a Neurology Specialist Examination in Spain

Pablo Ros-Arlanzón iD, Angel Perez-Sempere iD

JMIR Medical Education · 2024

DOI: 10.2196/56762 ↗

Kurz gefasst

Vergleichsstudie in der Zeitschrift JMIR Medical Education, 2024. Verfasst von Pablo Ros-Arlanzón und 1 weiteren Autor:innen. Beteiligte Einrichtung: Department of Neurology, Dr. Balmis General University Hospital, C/ Pintor Baeza, Nº 11, Alicante, 03010, Spain, 34 965933000. Hauptthemen laut MEDLINE: Artificial Intelligence, Neurologists, Neurology, Clinical Competence. Die Arbeit wurde bisher 9-mal zitiert (Crossref) und nennt 31 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

With the rapid advancement of artificial intelligence (AI) in various fields, evaluating its application in specialized medical contexts becomes crucial. ChatGPT, a large language model developed by OpenAI, has shown potential in diverse applications, including medicine.

Objective

This study aims to compare the performance of ChatGPT with that of attending neurologists in a real neurology specialist examination conducted in the Valencian Community, Spain, assessing the AI’s capabilities and limitations in medical knowledge.

Methods

We conducted a comparative analysis using the 2022 neurology specialist examination results from 120 neurologists and responses generated by ChatGPT versions 3.5 and 4. The examination consisted of 80 multiple-choice questions, with a focus on clinical neurology and health legislation. Questions were classified according to Bloom’s Taxonomy. Statistical analysis of performance, including the κ coefficient for response consistency, was performed.

Results

Human participants exhibited a median score of 5.91 (IQR: 4.93-6.76), with 32 neurologists failing to pass. ChatGPT-3.5 ranked 116th out of 122, answering 54.5% of questions correctly (score 3.94). ChatGPT-4 showed marked improvement, ranking 17th with 81.8% of correct answers (score 7.57), surpassing several human specialists. No significant variations were observed in the performance on lower-order questions versus higher-order questions. Additionally, ChatGPT-4 demonstrated increased interrater reliability, as reflected by a higher κ coefficient of 0.73, compared to ChatGPT-3.5’s coefficient of 0.69.

Conclusions

This study underscores the evolving capabilities of AI in medical knowledge assessment, particularly in specialized fields. ChatGPT-4’s performance, outperforming the median score of human participants in a rigorous neurology examination, represents a significant milestone in AI development, suggesting its potential as an effective tool in specialized medical education and assessment.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Artificial IntelligenceNeurologistsNeurologyClinical CompetenceHumansSpainEducational MeasurementSpecialization

Stichwörter der Autor:innen

ChatGPTOpenAIartificial intelligenceclinical decision-makingmedical educationmedical knowledge assessment

Autor:innen und Institutionen

  • Department of Neurology, Dr. Balmis General University Hospital, C/ Pintor Baeza, Nº 11, Alicante, 03010, Spain, 34 965933000.
  • Department of Neuroscience, Instituto de Investigación Sanitaria y Biomédica de Alicante, Alicante, Spain.
  • Department of Clinical Medicine, Miguel Hernández University, Alicante, Spain.

ORCID-Profile: Pablo Ros-Arlanzón, Angel Perez-Sempere

Zitieren

APA

Ros-Arlanzón, P., & Perez-Sempere, A. (2024). Evaluating AI Competence in Specialized Medicine: Comparative Analysis of ChatGPT and Neurologists in a Neurology Specialist Examination in Spain. JMIR Medical Education. https://doi.org/10.2196/56762

Vancouver

Ros-Arlanzón P, Perez-Sempere A. Evaluating AI Competence in Specialized Medicine: Comparative Analysis of ChatGPT and Neurologists in a Neurology Specialist Examination in Spain. JMIR Medical Education. 2024. doi:10.2196/56762

BibTeX
@article{RosArlanzn2024,
  author = {Ros-Arlanzón, Pablo and Perez-Sempere, Angel},
  title = {Evaluating AI Competence in Specialized Medicine: Comparative Analysis of ChatGPT and Neurologists in a Neurology Specialist Examination in Spain},
  journal = {JMIR Medical Education},
  year = {2024},
  doi = {10.2196/56762}
}
Kostenlos starten →