EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC 4.0

Towards Community-Based Evaluation of AI in Neurology: Development of a Headache Diagnosis Dataset for Large Language Models

Anika Zahn, Sebastian Strauss, Dorian Zwanzig

Studies in Health Technology and Informatics · 2025

DOI: 10.3233/shti251535 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Studies in Health Technology and Informatics, 2025. Verfasst von Anika Zahn und 2 weiteren Autor:innen. Beteiligte Einrichtung: Department of Neurology, University Medicine Greifswald, Greifswald, Germany. Hauptthemen laut MEDLINE: Artificial Intelligence, Neurology, Headache, Natural Language Processing. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Diagnosing headache disorders remains a clinical challenge due to the heterogeneity of headache phenotypes and the absence of objective biomarkers. This study presents a curated dataset of 50 clinical headache case examples, comprising both real (n = 34) and synthetic (n = 16) cases, categorized across 20 diagnoses according to ICHD-3 criteria. The dataset enables the evaluation of large language models (LLMs) for diagnostic accuracy in headache medicine. Three GPT-based models were tested using different prompting strategies, with diagnostic performance assessed at both diagnosis and group levels. Top-1 accuracy ranged from 24% to 63% at the diagnosis level and up to 92% at the group level. The results highlight the potential of LLMs in supporting differential diagnosis of headache disorders, while also emphasizing the need for further validation with larger, diverse datasets. Future efforts will focus on expanding real-world data through clinical collaborations and benchmarking LLMs against medical professionals to assess their utility in clinical decision-making.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Artificial IntelligenceNeurologyHeadacheNatural Language ProcessingHeadache DisordersLanguageHumansDiagnosis, DifferentialLarge Language Models

Stichwörter der Autor:innen

AIdata setheadache diagnosislarge language model

Autor:innen und Institutionen

  • Department of Neurology, University Medicine Greifswald, Greifswald, Germany.
  • Department of Computer Science, Communication and Economics, University of Applied Sciences for Technology and Economics, Berlin, Germany.

Zitieren

APA

Zahn, A., Strauss, S., & Zwanzig, D. (2025). Towards Community-Based Evaluation of AI in Neurology: Development of a Headache Diagnosis Dataset for Large Language Models. Studies in Health Technology and Informatics. https://doi.org/10.3233/shti251535

Vancouver

Zahn A, Strauss S, Zwanzig D. Towards Community-Based Evaluation of AI in Neurology: Development of a Headache Diagnosis Dataset for Large Language Models. Studies in Health Technology and Informatics. 2025. doi:10.3233/shti251535

BibTeX
@article{Zahn2025,
  author = {Zahn, Anika and Strauss, Sebastian and Zwanzig, Dorian},
  title = {Towards Community-Based Evaluation of AI in Neurology: Development of a Headache Diagnosis Dataset for Large Language Models},
  journal = {Studies in Health Technology and Informatics},
  year = {2025},
  doi = {10.3233/shti251535}
}
Kostenlos starten →