Evaluation of GPT ‐5, a Large Language Model, in Replicating German Clinical Practice Guideline Recommendations in Oral Oncology: A Cross‐Sectional Concordance Study
Journal of Oral Pathology & Medicine · 2026-08 · 55 (7) · 856-860
DOI: 10.1111/jop.70140 ↗Kurz gefasst
Leitlinie in der Zeitschrift Journal of Oral Pathology & Medicine, online erschienen am 12.04.2026. Verfasst von Julius Hirsch und 4 weiteren Autor:innen. Beteiligte Einrichtung: Private Oral and Maxillofacial Surgery Practice "Dr. Dr. Hirsch & Söhne", Ulm, Germany. Hauptthemen laut MEDLINE: Large Language Models, Practice Guidelines as Topic, Mouth Neoplasms. Die Arbeit nennt 13 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
Artificial intelligence (AI) technologies, particularly large language models (LLMs) such as ChatGPT, are increasingly utilised in medical education and clinical information retrieval. Nevertheless, their capacity to accurately reproduce recommendations from established clinical practice guidelines (CPGs) has not been thoroughly examined. The present study evaluated the concordance between responses generated by GPT‐5 and recommendations contained in German CPGs addressing oral potentially malignant disorders (OPMDs) and oral carcinomas (OCs).
Methods
A cross‐sectional analytical comparison was performed between GPT‐5 outputs and German CPG recommendations available as of October 2025. Individual guideline statements were entered verbatim into GPT‐5, which was asked to confirm or reject the statements. To assess methodological robustness, inverted versions of the same statements were additionally tested. GPT‐5 was accessed through the free version without internet connectivity to ensure that responses originated solely from the model's internal training data. Accuracy was defined as the proportion of correctly classified statements. Concordance between guideline content and model responses was quantified using Cohen's 𝝹.
Results
Two German CPGs comprising 111 recommendations were included: the S2k guideline for OPMDs (15 recommendations) and the S3 guideline for OCs (96 recommendations). GPT‐5 correctly affirmed all authentic recommendations and rejected all inverted statements. Agreement between guideline statements and GPT‐5 responses was perfect when the original recommendations were analysed (𝝹 = 1.0) and remained very high when both original and inverted statements were evaluated jointly (𝝹 = 0.96). The majority of references cited within the guidelines were published in English (> 93%) and originated from outside Germany (> 77%).
Conclusion
When guideline recommendations were presented verbatim, GPT‐5 demonstrated complete concordance with German oral oncology CPGs. These findings indicate that the model is capable of recognising and retrieving established guideline information. However, this experimental design evaluates recognition of existing statements rather than autonomous clinical reasoning. At present, LLMs should therefore be regarded primarily as educational and informational tools rather than a replacement for expert clinical judgement in oral oncology.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Private Oral and Maxillofacial Surgery Practice "Dr. Dr. Hirsch & Söhne", Ulm, Germany.
- Department of Medicine, Danube Private University, Krems an der Donau, Austria.
- Department of Oral and Maxillofacial Surgery, Faculty of Dentistry, Chulalongkorn University, Bangkok, Thailand.
- Department of Statistics, Chulalongkorn University Business School, Bangkok, Thailand.
- Department of Oral, Craniomaxillofacial and Plastic Surgery, Faculty of Medicine, Brandenburg Medical School Theodor Fontane, Brandenburg an der Havel, Germany.
- Department of Oral, Craniomaxillofacial Plastic, Head and Neck Surgery, Sana Hospital Offenbach, Academic Teaching Hospital of Goethe University of Frankfurt, Offenbach am Main, Germany.
ORCID-Profile: Julius Hirsch, Keskanya Subbalekha, Chatpong Tangmanee, Christian Stoll, Poramate Pitak-Arnnop
Zitieren
APA
Hirsch, J., Subbalekha, K., Tangmanee, C., Stoll, C., & Pitak-Arnnop, P. (2026). Evaluation of GPT ‐5, a Large Language Model, in Replicating German Clinical Practice Guideline Recommendations in Oral Oncology: A Cross‐Sectional Concordance Study. Journal of Oral Pathology & Medicine, 55(7), 856-860. https://doi.org/10.1111/jop.70140
Vancouver
Hirsch J, Subbalekha K, Tangmanee C, Stoll C, Pitak-Arnnop P. Evaluation of GPT ‐5, a Large Language Model, in Replicating German Clinical Practice Guideline Recommendations in Oral Oncology: A Cross‐Sectional Concordance Study. Journal of Oral Pathology & Medicine. 2026;55(7):856-860. doi:10.1111/jop.70140
BibTeX
@article{Hirsch2026,
author = {Hirsch, Julius and Subbalekha, Keskanya and Tangmanee, Chatpong and Stoll, Christian and Pitak-Arnnop, Poramate},
title = {Evaluation of GPT ‐5, a Large Language Model, in Replicating German Clinical Practice Guideline Recommendations in Oral Oncology: A Cross‐Sectional Concordance Study},
journal = {Journal of Oral Pathology & Medicine},
year = {2026},
volume = {55},
number = {7},
pages = {856--860},
doi = {10.1111/jop.70140}
}
EUVIMED