Large Language Models in Multidisciplinary Decision-Making for Hepatopancreatobiliary Oncology: Retrospective Comparative Feasibility Study
Journal of Medical Internet Research · 17.09.2026 · 28 · e101137-e101137
DOI: 10.2196/101137 ↗Kurz gefasst
Vergleichsstudie in der Zeitschrift Journal of Medical Internet Research, online erschienen am 17.09.2026. Verfasst von Sung Jun Jo und 16 weiteren Autor:innen. Beteiligte Einrichtung: Department of Surgery, CHA University Bundang Medical Center, 59 Yatap-ro, Bundang-gu, Seongnam, Gyeonggi-do, Republic of Korea, 82 031-780-5000. Hauptthemen laut MEDLINE: Large Language Models, Liver Neoplasms, Clinical Decision-Making, Decision Making. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 25 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
Hepatopancreatobiliary (HPB) malignancies require complex treatment planning that often relies on multidisciplinary team (MDT) discussions. Large language models (LLMs) have recently been explored for clinical decision support, but their performance within real-world multidisciplinary decision environments remains unclear. In particular, the stability of LLM-generated recommendations—that is, whether a model produces the same answer when given the same clinical input—has rarely been examined.
Objective
This study aimed to evaluate the stability of treatment recommendations generated by contemporary LLMs when identical HPB cases are queried repeatedly, and their concordance with the treatment decisions reached at an institutional MDT conference.
Methods
This retrospective study included consecutive cases discussed at a single-center HPB MDT conference between September 1, 2024, and August 31, 2025. Standardized clinical case summaries derived from preconference documentation were provided to 4 LLMs (GPT-4o, GPT-5.2, Gemini 3 Pro, and Claude Sonnet 4.5) through their consumer web interfaces. Each model recommended a treatment among predefined MDT treatment options, and identical queries were repeated 4 times in separate sessions. Stability was quantified as the discordance rate relative to the initial response and, without privileging any single query, as the mean pairwise agreement and Fleiss κ across the 4 iterations. Concordance with MDT decisions was assessed using both the initial and modal responses, together with Cohen κ and class-wise F 1 -scores.
Results
A total of 107 MDT cases were analyzed. Stability differed significantly across models ( P =.01). Gemini 3 Pro showed the lowest discordance rate (mean 12.8%, SD 2.3%) and the highest reference-free agreement (Fleiss κ=0.737), whereas GPT-4o showed the highest discordance rate (mean 30.2%, SD 6.5%) and the lowest agreement (Fleiss κ=0.430). Concordance with MDT decisions ranged from 48.6% to 72.9% using the initial response and from 66.3% to 74.5% using the modal response, and the highest-performing model differed between the 2 definitions. Class-wise F 1 -score was consistently lower for surgery (0.400‐0.520) than for chemotherapy (0.621‐0.836). Complete discordance occurred in 17 of 107 (15.9%) cases and in none of the 31 anatomically unresectable cases (Fisher exact test, P =.003). Recurrent or on-treatment disease (adjusted odds ratio [OR] 5.40, 95% CI 1.65-17.68; P =.005), pancreatic tumor location (adjusted OR 7.37, 95% CI 2.03-26.78; P =.002), and low MDT agreement level (adjusted OR 10.33, 95% CI 1.54-69.38; P =.016) were independently associated with complete discordance.
Conclusions
LLM-generated treatment recommendations demonstrated moderate alignment with MDT decisions in HPB oncology. Importantly, response stability varied substantially across models, indicating that concordance alone is insufficient for evaluating LLMs as clinical decision support tools. These findings suggest that LLMs may serve as a reasoning-support layer in MDT-like decision environments, but their response stability must be systematically characterized before clinical integration.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Surgery, CHA University Bundang Medical Center, 59 Yatap-ro, Bundang-gu, Seongnam, Gyeonggi-do, Republic of Korea, 82 031-780-5000.
- Department of Medical Oncology, CHA University Bundang Medical Center, Seongnam, Gyeonggi-do, Republic of Korea.
- Department of Gastroenterology, CHA University Bundang Medical Center, Seongnam, Gyeonggi-do, Republic of Korea.
- Department of Radiology, CHA University Bundang Medical Center, Seongnam, Gyeonggi-do, Republic of Korea.
- Department of Nuclear Medicine, CHA University Bundang Medical Center, Seongnam, Gyeonggi-do, Republic of Korea.
- Department of Radiation Oncology, CHA University Bundang Medical Center, Seongnam, Gyeonggi-do, Republic of Korea.
Zitieren
APA
Jo, S. J., Chong, E. H., Kang, I., Yang, S. J., Kang, B., Kim, J. S., Chon, H. J., Kwon, C. I., Sung, M. J., Shin, S. P., An, C., Lim, H. Y., Yu, J. S., Jang, S., Im, J. H., Ko, K. H., & Lee, S. H. (2026). Large Language Models in Multidisciplinary Decision-Making for Hepatopancreatobiliary Oncology: Retrospective Comparative Feasibility Study. Journal of Medical Internet Research, 28, e101137-e101137. https://doi.org/10.2196/101137
Vancouver
Jo SJ, Chong EH, Kang I, Yang SJ, Kang B, Kim JS, et al. Large Language Models in Multidisciplinary Decision-Making for Hepatopancreatobiliary Oncology: Retrospective Comparative Feasibility Study. Journal of Medical Internet Research. 2026;28:e101137-e101137. doi:10.2196/101137
BibTeX
@article{Jo2026,
author = {Jo, Sung Jun and Chong, Eui Hyuk and Kang, Incheon and Yang, Seok Jeong and Kang, Beodeul and Kim, Jung Sun and Chon, Hong Jae and Kwon, Chang-Il and Sung, Min Je and Shin, Suk Pyo and An, Chansik and Lim, Ho Yeong and Yu, Jeong-Sik and Jang, Sujin and Im, Jung Ho and Ko, Kwang Hyun and Lee, Sung Hwan},
title = {Large Language Models in Multidisciplinary Decision-Making for Hepatopancreatobiliary Oncology: Retrospective Comparative Feasibility Study},
journal = {Journal of Medical Internet Research},
year = {2026},
volume = {28},
pages = {e101137--e101137},
doi = {10.2196/101137}
}
EUVIMED