EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Development and Evaluation of a GPT4-Based Orofacial Pain Clinical Decision Support System

Charlotte Vueghs iD, Hamid Shakeri, Tara Renton, Frederic Van der Cruyssen

Diagnostics · 17.12.2024 · 14 (24) · 2835

DOI: 10.3390/diagnostics14242835 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Diagnostics, online erschienen am 17.12.2024. Verfasst von Charlotte Vueghs und 3 weiteren Autor:innen. Beteiligte Einrichtung: Department of Oral and Maxillofacial Surgery, University Hospitals Leuven, 3000 Leuven, Belgium. Die Arbeit wurde bisher 13-mal zitiert (Crossref) und nennt 48 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Orofacial pain (OFP) encompasses a complex array of conditions affecting the face, mouth, and jaws, often leading to significant diagnostic challenges and high rates of misdiagnosis. Artificial intelligence, particularly large language models like GPT4 (OpenAI, San Francisco, CA, USA), offers potential as a diagnostic aid in healthcare settings.

Objective

To evaluate the diagnostic accuracy of GPT4 in OFP cases as a clinical decision support system (CDSS) and compare its performance against treating clinicians, expert evaluators, medical students, and general practitioners.

Methods

A total of 100 anonymized patient case descriptions involving diverse OFP conditions were collected. GPT4 was prompted to generate primary and differential diagnoses for each case using the International Classification of Orofacial Pain (ICOP) criteria. Diagnoses were compared to gold-standard diagnoses established by treating clinicians, and a scoring system was used to assess accuracy at three hierarchical ICOP levels. A subset of 24 cases was also evaluated by two clinical experts, two final-year medical students, and two general practitioners for comparative analysis. Diagnostic performance and interrater reliability were calculated.

Results

GPT4 achieved the highest accuracy level (ICOP level 3) in 38% of cases, with an overall diagnostic performance score of 157 out of 300 points (52%). The model provided accurate differential diagnoses in 80% of cases (400 out of 500 points). In the subset of 24 cases, the model’s performance was comparable to non-expert human evaluators but was surpassed by clinical experts, who correctly diagnosed 54% of cases at level 3. GPT4 demonstrated high accuracy in specific categories, correctly diagnosing 81% of trigeminal neuralgia cases at level 3. Interrater reliability between GPT4 and human evaluators was low (κ = 0.219, p < 0.001), indicating variability in diagnostic agreement.

Conclusions

GPT4 shows promise as a CDSS for OFP by improving diagnostic accuracy and offering structured differential diagnoses. While not yet outperforming expert clinicians, GPT4 can augment diagnostic workflows, particularly in primary care or educational settings. Effective integration into clinical practice requires adherence to rigorous guidelines, thorough validation, and ongoing professional oversight to ensure patient safety and diagnostic reliability.

Schlagwörter

Stichwörter der Autor:innen

GPT4clinical decision support systemdevelopmentlarge language modelvalidation

Autor:innen und Institutionen

  • Department of Oral and Maxillofacial Surgery, University Hospitals Leuven, 3000 Leuven, Belgium.
  • Department of Oral Surgery, King's College London Dental Institute, London SE5 9RW, UK.
  • OMFS-IMPATH Research Group, KU Leuven, 3000 Leuven, Belgium.

ORCID-Profile: Charlotte Vueghs

Zitieren

APA

Vueghs, C., Shakeri, H., Renton, T., & Van der Cruyssen, F. (2024). Development and Evaluation of a GPT4-Based Orofacial Pain Clinical Decision Support System. Diagnostics, 14(24), 2835. https://doi.org/10.3390/diagnostics14242835

Vancouver

Vueghs C, Shakeri H, Renton T, Van der Cruyssen F. Development and Evaluation of a GPT4-Based Orofacial Pain Clinical Decision Support System. Diagnostics. 2024;14(24):2835. doi:10.3390/diagnostics14242835

BibTeX
@article{Vueghs2024,
  author = {Vueghs, Charlotte and Shakeri, Hamid and Renton, Tara and Van der Cruyssen, Frederic},
  title = {Development and Evaluation of a GPT4-Based Orofacial Pain Clinical Decision Support System},
  journal = {Diagnostics},
  year = {2024},
  volume = {14},
  number = {24},
  pages = {2835},
  doi = {10.3390/diagnostics14242835}
}
Kostenlos starten →