The illusion of competence: Evaluating the clinical reasoning of large language models in pediatric gastroenterology
Journal of Pediatric Gastroenterology and Nutrition · 16.07.2026
DOI: 10.1002/jpn3.70502 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Pediatric Gastroenterology and Nutrition, online erschienen am 16.07.2026. Verfasst von Yasin Maruf Ergen und 3 weiteren Autor:innen. Beteiligte Einrichtung: Department of Pediatrics/Division of Pediatric Gastroenterology, Gülhane Training and Research Hospital, Ankara, Turkey. Hauptthemen laut MEDLINE: Large Language Models, Clinical Competence, Pediatrics, Gastroenterology. Die Arbeit nennt 19 Literaturangaben. Gefördert von: None.
Abstract (Originalsprache)
Objective
To evaluate the diagnostic accuracy and clinical reasoning of three frontier large language models (LLMs) across standardized pediatric gastroenterology, hepatology, and nutrition (PGHN) clinical vignettes.
Methods
In this cross‐sectional study, 25 fictional PGHN vignettes were developed by one board‐certified pediatric gastroenterologist and evaluated using three LLMs: Gemini 3.1 Pro, ChatGPT 5.4 Thinking, and Claude Sonnet 4.6 Extended. A conditional two‐step prompting protocol was applied. Three blinded, PGHN‐certified co‐authors independently scored responses using a structured instrument covering four domains (diagnostic accuracy, management, patient safety, reference quality; 0–2 each) and a global quality of clinical reasoning (QCR) score (1–5 Likert scale). Interobserver reliability was assessed using the intraclass correlation coefficient (ICC). Group differences were analyzed using the Kruskal–Wallis H test with Dunn post‐hoc correction.
Results
Interobserver reliability was excellent (ICC: 0.98 for domains; 0.87 for QCR). All three models achieved perfect diagnostic accuracy (median 2.00, interquartile range: 2.00–2.00). Statistically significant intermodel differences were identified in reference quality ( p = 0.048) and QCR ( p = 0.049). Claude achieved significantly higher QCR scores than ChatGPT ( p = 0.043) and demonstrated the highest overall reference quality. Qualitative analysis revealed critical pharmacological dosing errors, contextual blindness, temporal obsolescence, and a high frequency of fabricated citations.
Conclusion
While current LLMs demonstrate good diagnostic pattern recognition in PGHN, reproducible and potentially life‐threatening failures in pharmacological reasoning and reference accuracy create a dangerous illusion of competence. These findings suggest that LLMs may support differential diagnosis brainstorming in PGHN but do not establish their safety in real‐world clinical scenarios.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Pediatrics/Division of Pediatric Gastroenterology, Gülhane Training and Research Hospital, Ankara, Turkey.
ORCID-Profile: Yasin Maruf Ergen, Selçuk Teke, Edibe Gözde Başaran, Necati Balamtekin
Förderung
- None
Zitieren
APA
Ergen, Y. M., Teke, S., Başaran, E. G., & Balamtekin, N. (2026). The illusion of competence: Evaluating the clinical reasoning of large language models in pediatric gastroenterology. Journal of Pediatric Gastroenterology and Nutrition. https://doi.org/10.1002/jpn3.70502
Vancouver
Ergen YM, Teke S, Başaran EG, Balamtekin N. The illusion of competence: Evaluating the clinical reasoning of large language models in pediatric gastroenterology. Journal of Pediatric Gastroenterology and Nutrition. 2026. doi:10.1002/jpn3.70502
BibTeX
@article{Ergen2026,
author = {Ergen, Yasin Maruf and Teke, Selçuk and Başaran, Edibe Gözde and Balamtekin, Necati},
title = {The illusion of competence: Evaluating the clinical reasoning of large language models in pediatric gastroenterology},
journal = {Journal of Pediatric Gastroenterology and Nutrition},
year = {2026},
doi = {10.1002/jpn3.70502}
}
EUVIMED