Large Language Models in Neurological Practice: Real-World Study
Journal of Medical Internet Research · 2025
DOI: 10.2196/73212 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Medical Internet Research, 2025. Verfasst von Natale Vincenzo Maiorana und 9 weiteren Autor:innen. Beteiligte Einrichtung: Aldo Ravelli Center for Neurotechnology and Experimental Brain Therapeutics, Department of Health Sciences, University of Milan, Via Antonio di Rudin…. Hauptthemen laut MEDLINE: Neurology, Nervous System Diseases, Language. Die Arbeit wurde bisher 14-mal zitiert (Crossref) und nennt 47 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
Large language models (LLMs) such as ChatGPT (OpenAI) and Gemini (Google) are increasingly explored for their potential in medical diagnostics, including neurology. Their real-world applicability remains inadequately assessed, particularly in clinical workflows where nuanced decision-making is required.
Objective
This study aims to evaluate the diagnostic accuracy and appropriateness of clinical recommendations provided by not-specifically-trained, freely available ChatGPT and Gemini, compared to neurologists, using real-world clinical cases.
Methods
This study consisted of an experimental evaluation of LLMs’ diagnostic performance presenting real-world neurology cases to ChatGPT and Gemini, comparing their performance with that of clinical neurologists. The study was conducted simulating a first visit using information from anonymized patient records from the Neurology Department of the ASST Santi Paolo e Carlo Hospital, ensuring a real-world clinical context. The study involved a cohort of 28 anonymized patient cases covering a range of neurological conditions and diagnostic complexities representative of daily clinical practice. The primary outcome was diagnostic accuracy of both neurologists and LLMs, defined as concordance with discharge diagnoses. Secondary outcomes included the appropriateness of recommended diagnostic tests, interrater agreement, and the extent of additional prompting required for accurate responses.
Results
Neurologists achieved a diagnostic accuracy of 75%, outperforming ChatGPT (54%) and Gemini (46%). Both LLMs demonstrated limitations in nuanced clinical reasoning and overprescribed diagnostic tests in 17%‐25% of cases. In addition, complex or ambiguous cases required further prompting to refine artificial intelligence–generated responses. Interrater reliability analysis using Fleiss Kappa showed a moderate-to-substantial level of agreement among raters (κ=0.47, SE 0.077; z =6.14, P
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Aldo Ravelli Center for Neurotechnology and Experimental Brain Therapeutics, Department of Health Sciences, University of Milan, Via Antonio di Rudinì, 8, Milan, 20142, Italy, 39 02 50323233.
- Clinical Neurology Unit, Department of Health Sciences, Azienda Socio-Sanitaria Territoriale Santi Paolo e Carlo, University of Milan, Milan, Italy.
Zitieren
APA
Maiorana, N. V., Marceglia, S., Treddenti, M., Tosi, M., Guidetti, M., Creta, M. F., Bocci, T., Oliveri, S., Martinelli Boneschi, F., & Priori, A. (2025). Large Language Models in Neurological Practice: Real-World Study. Journal of Medical Internet Research. https://doi.org/10.2196/73212
Vancouver
Maiorana NV, Marceglia S, Treddenti M, Tosi M, Guidetti M, Creta MF, et al. Large Language Models in Neurological Practice: Real-World Study. Journal of Medical Internet Research. 2025. doi:10.2196/73212
BibTeX
@article{Maiorana2025,
author = {Maiorana, Natale Vincenzo and Marceglia, Sara and Treddenti, Mauro and Tosi, Mattia and Guidetti, Matteo and Creta, Maria Francesca and Bocci, Tommaso and Oliveri, Serena and Martinelli Boneschi, Filippo and Priori, Alberto},
title = {Large Language Models in Neurological Practice: Real-World Study},
journal = {Journal of Medical Internet Research},
year = {2025},
doi = {10.2196/73212}
}
EUVIMED