EUVIMEDEUVIMED entdecken
VergleichsstudieOpen Access · CC BY-NC 4.0

Comparative diagnostic agreement of a supervised machine learning model and a general-purpose, zero-shot, non-domain-adapted large language model for classifying headache disorders using structured questionnaires

Masahito Katsuki iD, Kieran Moran iD, Siobhán O'Connor iD, Tomás Ward, Yutaro Fuse, Omid Kohandel Gargari iD, Marina Romozzi iD, Alicia Gonzalez-Martinez iD, Miguel Á Huerta iD, Woo-Seok Ha iD, Jackson Ts Cheung, Yasuhiko Matsumori iD

Cephalalgia · 2026-04 · 46 (4)

DOI: 10.1177/03331024261441574 ↗

Kurz gefasst

Vergleichsstudie in der Zeitschrift Cephalalgia, online erschienen am 17.04.2026. Verfasst von Masahito Katsuki und 11 weiteren Autor:innen. Beteiligte Einrichtung: Physical Education and Health Center, Nagaoka University of Technology, Nagaoka, Japan. Hauptthemen laut MEDLINE: Headache Disorders, Supervised Machine Learning. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 40 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0). Gefördert von: H2020 Marie Skłodowska-Curie Actions, Insight Research Ireland Centre for Data Analytics.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Accurate diagnosis of headache disorders is essential in clinical practice. Supervised machine learning models trained on structured clinical data have shown good performance, whereas the diagnostic ability of large language models (LLMs) for headache disorders has not been evaluated. This study compared a validated machine learning classifier with a general-purpose, zero-shot, non-domain-adapted LLM using the same structured patient questionnaire data, focusing on their agreement with specialist-confirmed diagnoses as the ground truth. This study was designed to reflect current real-world use scenarios, in which clinicians may apply off-the-shelf LLMs for diagnostic purposes without few-shot prompting, domain-specific fine-tuning, or adaptation, rather than to assess the theoretical upper limits of LLM capabilities.

Methods

We analyzed 1818 patients from an independent hold-out test cohort who completed a 22-item structured headache questionnaire and received specialist-confirmed diagnoses. A previously developed machine learning model and a general-purpose, non-domain-adapted LLM (GPT-4.1 with zero-shot prompting) each generated five-class International Classification of Headache Disorders, 3rd edition (ICHD-3)-based predictions: migraine and/or medication-overuse headache (MOH), tension-type headache (TTH), trigeminal autonomic cephalalgias (TACs), other primary headache disorders, and secondary headaches. Agreement with the specialist's diagnosis and diagnostic performance metrics were calculated. Class-wise sensitivity and specificity were compared using McNemar's test.

Results

The machine learning classifier showed significantly higher diagnostic agreement with the specialist than the LLM (Cohen's κ: 0.46 vs. 0.26; 95% confidence interval of the difference: 0.15–0.25). Although the LLM showed slightly higher macro-averaged sensitivity (balanced accuracy) than the machine learning model, the machine learning classifier showed higher macro-averaged precision, specificity, and F-value. Class-wise analysis showed that the machine learning model demonstrated greater sensitivity for migraine and/or MOH and secondary headaches, while the LLM showed higher sensitivity for TTH. Regarding specificity, the machine learning model outperformed the LLM in TTH, TACs, and other primary headache disorders, whereas the LLM showed higher specificity only for migraine and/or MOH.

Conclusions

A supervised machine learning model trained on real-world clinical data showed better agreement with a specialist-confirmed diagnosis than a general-purpose, zero-shot, non-domain-adapted LLM. These findings indicate that, in its current off-the-shelf configuration under this experimental setting, the diagnostic agreement between a general-purpose LLM and specialists can be limited for headache disorders.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Headache DisordersSupervised Machine LearningHumansLarge Language ModelsSurveys and QuestionnairesFemaleMaleAdultMiddle AgedClassification AlgorithmsPredictive Learning Models

Stichwörter der Autor:innen

ChatGPTartificial intelligence (AI)headache diagnosislarge language models (LLMs)machine learning (ML)

Autor:innen und Institutionen

  • Physical Education and Health Center, Nagaoka University of Technology, Nagaoka, Japan.
  • School of Health and Human Performance, Dublin City University, Dublin, Ireland.
  • Insight Research Ireland Centre for Data Analytics, Dublin City University, Dublin, Ireland.
  • Department of Biostatistics, Graduate School of Medicine, Saitama Medical University, Saitama, Japan.
  • Insight Research Ireland Centre for Data Analytics, Maynooth University, Kildare, Ireland.
  • Department of Sport Science and Nutrition, Maynooth University, Kildare, Ireland.

ORCID-Profile: Masahito Katsuki, Kieran Moran, Siobhán O'Connor, Omid Kohandel Gargari, Marina Romozzi, Alicia Gonzalez-Martinez, Miguel Á Huerta, Woo-Seok Ha, Yasuhiko Matsumori

Förderung

  • H2020 Marie Skłodowska-Curie Actions
  • Insight Research Ireland Centre for Data Analytics

Zitieren

APA

Katsuki, M., Moran, K., O'Connor, S., Ward, T., Fuse, Y., Gargari, O. K., Romozzi, M., Gonzalez-Martinez, A., Huerta, M. Á., Ha, W. S., Cheung, J. T., & Matsumori, Y. (2026). Comparative diagnostic agreement of a supervised machine learning model and a general-purpose, zero-shot, non-domain-adapted large language model for classifying headache disorders using structured questionnaires. Cephalalgia, 46(4). https://doi.org/10.1177/03331024261441574

Vancouver

Katsuki M, Moran K, O'Connor S, Ward T, Fuse Y, Gargari OK, et al. Comparative diagnostic agreement of a supervised machine learning model and a general-purpose, zero-shot, non-domain-adapted large language model for classifying headache disorders using structured questionnaires. Cephalalgia. 2026;46(4). doi:10.1177/03331024261441574

BibTeX
@article{Katsuki2026,
  author = {Katsuki, Masahito and Moran, Kieran and O'Connor, Siobhán and Ward, Tomás and Fuse, Yutaro and Gargari, Omid Kohandel and Romozzi, Marina and Gonzalez-Martinez, Alicia and Huerta, Miguel Á and Ha, Woo-Seok and Cheung, Jackson Ts and Matsumori, Yasuhiko},
  title = {Comparative diagnostic agreement of a supervised machine learning model and a general-purpose, zero-shot, non-domain-adapted large language model for classifying headache disorders using structured questionnaires},
  journal = {Cephalalgia},
  year = {2026},
  volume = {46},
  number = {4},
  doi = {10.1177/03331024261441574}
}
Kostenlos starten →