EUVIMEDEUVIMED entdecken
Crossref

The illusion of competence: Evaluating the clinical reasoning of large language models in pediatric gastroenterology

Yasin Maruf Ergen iD, Selçuk Teke iD, Edibe Gözde Başaran iD, Necati Balamtekin iD

Journal of Pediatric Gastroenterology and Nutrition · 16.07.2026

DOI: 10.1002/jpn3.70502 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Pediatric Gastroenterology and Nutrition, online erschienen am 16.07.2026. Verfasst von Yasin Maruf Ergen und 3 weiteren Autor:innen. Beteiligte Einrichtung: Department of Pediatrics/Division of Pediatric Gastroenterology, Gülhane Training and Research Hospital, Ankara, Turkey. Hauptthemen laut MEDLINE: Large Language Models, Clinical Competence, Pediatrics, Gastroenterology. Die Arbeit nennt 19 Literaturangaben. Gefördert von: None.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objective

To evaluate the diagnostic accuracy and clinical reasoning of three frontier large language models (LLMs) across standardized pediatric gastroenterology, hepatology, and nutrition (PGHN) clinical vignettes.

Methods

In this cross‐sectional study, 25 fictional PGHN vignettes were developed by one board‐certified pediatric gastroenterologist and evaluated using three LLMs: Gemini 3.1 Pro, ChatGPT 5.4 Thinking, and Claude Sonnet 4.6 Extended. A conditional two‐step prompting protocol was applied. Three blinded, PGHN‐certified co‐authors independently scored responses using a structured instrument covering four domains (diagnostic accuracy, management, patient safety, reference quality; 0–2 each) and a global quality of clinical reasoning (QCR) score (1–5 Likert scale). Interobserver reliability was assessed using the intraclass correlation coefficient (ICC). Group differences were analyzed using the Kruskal–Wallis H test with Dunn post‐hoc correction.

Results

Interobserver reliability was excellent (ICC: 0.98 for domains; 0.87 for QCR). All three models achieved perfect diagnostic accuracy (median 2.00, interquartile range: 2.00–2.00). Statistically significant intermodel differences were identified in reference quality ( p = 0.048) and QCR ( p = 0.049). Claude achieved significantly higher QCR scores than ChatGPT ( p = 0.043) and demonstrated the highest overall reference quality. Qualitative analysis revealed critical pharmacological dosing errors, contextual blindness, temporal obsolescence, and a high frequency of fabricated citations.

Conclusion

While current LLMs demonstrate good diagnostic pattern recognition in PGHN, reproducible and potentially life‐threatening failures in pharmacological reasoning and reference accuracy create a dangerous illusion of competence. These findings suggest that LLMs may support differential diagnosis brainstorming in PGHN but do not establish their safety in real‐world clinical scenarios.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Large Language ModelsClinical CompetencePediatricsGastroenterologyClinical ReasoningHumansCross-Sectional StudiesReproducibility of ResultsObserver VariationChild

Stichwörter der Autor:innen

artificial intelligenceclinical decision supportguideline concordancehallucinationreference hallucination

Autor:innen und Institutionen

  • Department of Pediatrics/Division of Pediatric Gastroenterology, Gülhane Training and Research Hospital, Ankara, Turkey.

ORCID-Profile: Yasin Maruf Ergen, Selçuk Teke, Edibe Gözde Başaran, Necati Balamtekin

Förderung

  • None

Zitieren

APA

Ergen, Y. M., Teke, S., Başaran, E. G., & Balamtekin, N. (2026). The illusion of competence: Evaluating the clinical reasoning of large language models in pediatric gastroenterology. Journal of Pediatric Gastroenterology and Nutrition. https://doi.org/10.1002/jpn3.70502

Vancouver

Ergen YM, Teke S, Başaran EG, Balamtekin N. The illusion of competence: Evaluating the clinical reasoning of large language models in pediatric gastroenterology. Journal of Pediatric Gastroenterology and Nutrition. 2026. doi:10.1002/jpn3.70502

BibTeX
@article{Ergen2026,
  author = {Ergen, Yasin Maruf and Teke, Selçuk and Başaran, Edibe Gözde and Balamtekin, Necati},
  title = {The illusion of competence: Evaluating the clinical reasoning of large language models in pediatric gastroenterology},
  journal = {Journal of Pediatric Gastroenterology and Nutrition},
  year = {2026},
  doi = {10.1002/jpn3.70502}
}
Kostenlos starten →