EUVIMEDEUVIMED entdecken
LeitlinieOpen Access

Evaluation of GPT ‐5, a Large Language Model, in Replicating German Clinical Practice Guideline Recommendations in Oral Oncology: A Cross‐Sectional Concordance Study

Julius Hirsch iD, Keskanya Subbalekha iD, Chatpong Tangmanee iD, Christian Stoll iD, Poramate Pitak-Arnnop iD

Journal of Oral Pathology & Medicine · 2026-08 · 55 (7) · 856-860

DOI: 10.1111/jop.70140 ↗

Kurz gefasst

Leitlinie in der Zeitschrift Journal of Oral Pathology & Medicine, online erschienen am 12.04.2026. Verfasst von Julius Hirsch und 4 weiteren Autor:innen. Beteiligte Einrichtung: Private Oral and Maxillofacial Surgery Practice "Dr. Dr. Hirsch & Söhne", Ulm, Germany. Hauptthemen laut MEDLINE: Large Language Models, Practice Guidelines as Topic, Mouth Neoplasms. Die Arbeit nennt 13 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Artificial intelligence (AI) technologies, particularly large language models (LLMs) such as ChatGPT, are increasingly utilised in medical education and clinical information retrieval. Nevertheless, their capacity to accurately reproduce recommendations from established clinical practice guidelines (CPGs) has not been thoroughly examined. The present study evaluated the concordance between responses generated by GPT‐5 and recommendations contained in German CPGs addressing oral potentially malignant disorders (OPMDs) and oral carcinomas (OCs).

Methods

A cross‐sectional analytical comparison was performed between GPT‐5 outputs and German CPG recommendations available as of October 2025. Individual guideline statements were entered verbatim into GPT‐5, which was asked to confirm or reject the statements. To assess methodological robustness, inverted versions of the same statements were additionally tested. GPT‐5 was accessed through the free version without internet connectivity to ensure that responses originated solely from the model's internal training data. Accuracy was defined as the proportion of correctly classified statements. Concordance between guideline content and model responses was quantified using Cohen's 𝝹.

Results

Two German CPGs comprising 111 recommendations were included: the S2k guideline for OPMDs (15 recommendations) and the S3 guideline for OCs (96 recommendations). GPT‐5 correctly affirmed all authentic recommendations and rejected all inverted statements. Agreement between guideline statements and GPT‐5 responses was perfect when the original recommendations were analysed (𝝹 = 1.0) and remained very high when both original and inverted statements were evaluated jointly (𝝹 = 0.96). The majority of references cited within the guidelines were published in English (> 93%) and originated from outside Germany (> 77%).

Conclusion

When guideline recommendations were presented verbatim, GPT‐5 demonstrated complete concordance with German oral oncology CPGs. These findings indicate that the model is capable of recognising and retrieving established guideline information. However, this experimental design evaluates recognition of existing statements rather than autonomous clinical reasoning. At present, LLMs should therefore be regarded primarily as educational and informational tools rather than a replacement for expert clinical judgement in oral oncology.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Large Language ModelsPractice Guidelines as TopicMouth NeoplasmsCross-Sectional StudiesHumansGermanyGenerative Artificial Intelligence

Stichwörter der Autor:innen

GPT‐5artificial intelligenceclinical practice guidelineoral oncology

Autor:innen und Institutionen

  • Private Oral and Maxillofacial Surgery Practice "Dr. Dr. Hirsch & Söhne", Ulm, Germany.
  • Department of Medicine, Danube Private University, Krems an der Donau, Austria.
  • Department of Oral and Maxillofacial Surgery, Faculty of Dentistry, Chulalongkorn University, Bangkok, Thailand.
  • Department of Statistics, Chulalongkorn University Business School, Bangkok, Thailand.
  • Department of Oral, Craniomaxillofacial and Plastic Surgery, Faculty of Medicine, Brandenburg Medical School Theodor Fontane, Brandenburg an der Havel, Germany.
  • Department of Oral, Craniomaxillofacial Plastic, Head and Neck Surgery, Sana Hospital Offenbach, Academic Teaching Hospital of Goethe University of Frankfurt, Offenbach am Main, Germany.

ORCID-Profile: Julius Hirsch, Keskanya Subbalekha, Chatpong Tangmanee, Christian Stoll, Poramate Pitak-Arnnop

Zitieren

APA

Hirsch, J., Subbalekha, K., Tangmanee, C., Stoll, C., & Pitak-Arnnop, P. (2026). Evaluation of GPT ‐5, a Large Language Model, in Replicating German Clinical Practice Guideline Recommendations in Oral Oncology: A Cross‐Sectional Concordance Study. Journal of Oral Pathology & Medicine, 55(7), 856-860. https://doi.org/10.1111/jop.70140

Vancouver

Hirsch J, Subbalekha K, Tangmanee C, Stoll C, Pitak-Arnnop P. Evaluation of GPT ‐5, a Large Language Model, in Replicating German Clinical Practice Guideline Recommendations in Oral Oncology: A Cross‐Sectional Concordance Study. Journal of Oral Pathology & Medicine. 2026;55(7):856-860. doi:10.1111/jop.70140

BibTeX
@article{Hirsch2026,
  author = {Hirsch, Julius and Subbalekha, Keskanya and Tangmanee, Chatpong and Stoll, Christian and Pitak-Arnnop, Poramate},
  title = {Evaluation of GPT ‐5, a Large Language Model, in Replicating German Clinical Practice Guideline Recommendations in Oral Oncology: A Cross‐Sectional Concordance Study},
  journal = {Journal of Oral Pathology & Medicine},
  year = {2026},
  volume = {55},
  number = {7},
  pages = {856--860},
  doi = {10.1111/jop.70140}
}
Kostenlos starten →