EUVIMEDEUVIMED entdecken
Open Access

Large Language Models in Neurological Practice: Real-World Study

Natale Vincenzo Maiorana, Sara Marceglia, Mauro Treddenti, Mattia Tosi, Matteo Guidetti, Maria Francesca Creta, Tommaso Bocci, Serena Oliveri, Filippo Martinelli Boneschi, Alberto Priori

Journal of Medical Internet Research · 2025

DOI: 10.2196/73212 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Medical Internet Research, 2025. Verfasst von Natale Vincenzo Maiorana und 9 weiteren Autor:innen. Beteiligte Einrichtung: Aldo Ravelli Center for Neurotechnology and Experimental Brain Therapeutics, Department of Health Sciences, University of Milan, Via Antonio di Rudin…. Hauptthemen laut MEDLINE: Neurology, Nervous System Diseases, Language. Die Arbeit wurde bisher 14-mal zitiert (Crossref) und nennt 47 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Large language models (LLMs) such as ChatGPT (OpenAI) and Gemini (Google) are increasingly explored for their potential in medical diagnostics, including neurology. Their real-world applicability remains inadequately assessed, particularly in clinical workflows where nuanced decision-making is required.

Objective

This study aims to evaluate the diagnostic accuracy and appropriateness of clinical recommendations provided by not-specifically-trained, freely available ChatGPT and Gemini, compared to neurologists, using real-world clinical cases.

Methods

This study consisted of an experimental evaluation of LLMs’ diagnostic performance presenting real-world neurology cases to ChatGPT and Gemini, comparing their performance with that of clinical neurologists. The study was conducted simulating a first visit using information from anonymized patient records from the Neurology Department of the ASST Santi Paolo e Carlo Hospital, ensuring a real-world clinical context. The study involved a cohort of 28 anonymized patient cases covering a range of neurological conditions and diagnostic complexities representative of daily clinical practice. The primary outcome was diagnostic accuracy of both neurologists and LLMs, defined as concordance with discharge diagnoses. Secondary outcomes included the appropriateness of recommended diagnostic tests, interrater agreement, and the extent of additional prompting required for accurate responses.

Results

Neurologists achieved a diagnostic accuracy of 75%, outperforming ChatGPT (54%) and Gemini (46%). Both LLMs demonstrated limitations in nuanced clinical reasoning and overprescribed diagnostic tests in 17%‐25% of cases. In addition, complex or ambiguous cases required further prompting to refine artificial intelligence–generated responses. Interrater reliability analysis using Fleiss Kappa showed a moderate-to-substantial level of agreement among raters (κ=0.47, SE 0.077; z =6.14, P

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

NeurologyNervous System DiseasesLanguageHumansLarge Language Models

Stichwörter der Autor:innen

ChatGPTGeminiartificial intelligenceclinical practicelarge language modelneurology

Autor:innen und Institutionen

  • Aldo Ravelli Center for Neurotechnology and Experimental Brain Therapeutics, Department of Health Sciences, University of Milan, Via Antonio di Rudinì, 8, Milan, 20142, Italy, 39 02 50323233.
  • Clinical Neurology Unit, Department of Health Sciences, Azienda Socio-Sanitaria Territoriale Santi Paolo e Carlo, University of Milan, Milan, Italy.

Zitieren

APA

Maiorana, N. V., Marceglia, S., Treddenti, M., Tosi, M., Guidetti, M., Creta, M. F., Bocci, T., Oliveri, S., Martinelli Boneschi, F., & Priori, A. (2025). Large Language Models in Neurological Practice: Real-World Study. Journal of Medical Internet Research. https://doi.org/10.2196/73212

Vancouver

Maiorana NV, Marceglia S, Treddenti M, Tosi M, Guidetti M, Creta MF, et al. Large Language Models in Neurological Practice: Real-World Study. Journal of Medical Internet Research. 2025. doi:10.2196/73212

BibTeX
@article{Maiorana2025,
  author = {Maiorana, Natale Vincenzo and Marceglia, Sara and Treddenti, Mauro and Tosi, Mattia and Guidetti, Matteo and Creta, Maria Francesca and Bocci, Tommaso and Oliveri, Serena and Martinelli Boneschi, Filippo and Priori, Alberto},
  title = {Large Language Models in Neurological Practice: Real-World Study},
  journal = {Journal of Medical Internet Research},
  year = {2025},
  doi = {10.2196/73212}
}
Kostenlos starten →