Development of a Framework for Evaluating Large Language Model Safety and Reliability: a Proof-of-Concept Evaluation
Journal of Medical Systems · 23.09.2026 · 50 (1)
DOI: 10.1007/s10916-026-02459-1 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Medical Systems, online erschienen am 23.09.2026. Verfasst von Fangyan Liu und 6 weiteren Autor:innen. Beteiligte Einrichtung: Department of Emergency, National Clinical Research Center for Geriatric Diseases, Xuanwu Hospital of Capital Medical University, Beijing, 100053, Ch…. Hauptthemen laut MEDLINE: Large Language Models, Decision Support Systems, Clinical. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 35 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Medical Science Research Project of Hebei, Science and Technology Innovation Program of Xiong'an New Area, National Science and Technology Major Project.
Abstract (Originalsprache)
Large language models (LLMs) are entering clinical decision support faster than methodology can characterise their safety. Aggregate accuracy treats all errors as interchangeable and cannot support safe deployment under Software as a Medical Device (SaMD) and EU AI Act frameworks. To develop and demonstrate a framework for evaluating large language model safety and reliability using catastrophic-failure frequency and response reproducibility, incorporating a pre-specified error taxonomy, difficulty-stratified analysis, and two-layer response consistency. The framework was applied to 54 diagnostically challenging emergency cases from non-public institutional records. Six LLMs were accessed via application programming interfaces (APIs) (text-only, zero-shot, defaults; August–December 2025) and queried three times each, producing 972 physician-scored responses. Aggregate scores concealed heterogeneity: anchoring-bias failure (score ≤ 3) varied nearly four-fold (6.9%–26.4%) and rare-disease recognition six-fold (6.1%–37.9%); catastrophic (dangerous-recommendation, score ≤ 2) rates were two-tiered (1.5% for the safest two vs. 6.0% for the rest; p < 0.001), though within-tier ranks were inseparable (2–12 per model). Despite a middle-tier mean, GPT-5.1 was least reproducible (within-case SD 2.69) and fell within the higher catastrophic-failure tier; across models, 31 of 34 dangerous combinations were stochastic, not systematic—a failure mode hidden by aggregate metrics. Open-source DeepSeek R1 was statistically equivalent to Gemini 3 Pro within a 1.5-point margin by two one-sided tests (TOST: Δ − 0.09; 90% CI − 0.47 to 0.30; p < 0.001). In this single-centre proof-of-concept, aggregate accuracy was insufficient for safety characterisation: models with indistinguishable mean accuracy carried different catastrophic-failure tiers and reproducibility profiles. Error-type and response-consistency profiling may inform model selection, ensemble design, and conformity assessment.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Emergency, National Clinical Research Center for Geriatric Diseases, Xuanwu Hospital of Capital Medical University, Beijing, 100053, China.
- Department of Emergency, Xiongan Xuanwu Hospital, No. 1, Xili Road, Xiongan New Area, Hebei, 070001, China.
- Information Center, National Clinical Research Center for Geriatric Diseases, Xuanwu Hospital of Capital Medical University, Beijing, 100053, China.
- Information Center, Xiongan Xuanwu Hospital, No. 1, Xili Road, Xiongan New Area, Hebei, 070001, China.
- Department of General Surgery, Xuanwu Hospital of Capital Medical University, No. 45, Changchun Street, Beijing, 100053, China. lijia@xwh.ccmu.edu.cn.
- Department of General Surgery, Xiongan Xuanwu Hospital, No. 1, Xili Road, Xiongan New Area, Hebei, 070001, China. lijia@xwh.ccmu.edu.cn.
Förderung
- Medical Science Research Project of Hebei
- Science and Technology Innovation Program of Xiong'an New Area
- National Science and Technology Major Project
- Science and Technology Innovation Program of Xiong’an New Area
Zitieren
APA
Liu, F., Liu, Z., Fei, X., He, J., Xing, J., Li, J., & Chan, P. (2026). Development of a Framework for Evaluating Large Language Model Safety and Reliability: a Proof-of-Concept Evaluation. Journal of Medical Systems, 50(1). https://doi.org/10.1007/s10916-026-02459-1
Vancouver
Liu F, Liu Z, Fei X, He J, Xing J, Li J, et al. Development of a Framework for Evaluating Large Language Model Safety and Reliability: a Proof-of-Concept Evaluation. Journal of Medical Systems. 2026;50(1). doi:10.1007/s10916-026-02459-1
BibTeX
@article{Liu2026,
author = {Liu, Fangyan and Liu, Zhi and Fei, Xiaolu and He, Jingyu and Xing, Jin and Li, Jia and Chan, Piu},
title = {Development of a Framework for Evaluating Large Language Model Safety and Reliability: a Proof-of-Concept Evaluation},
journal = {Journal of Medical Systems},
year = {2026},
volume = {50},
number = {1},
doi = {10.1007/s10916-026-02459-1}
}
EUVIMED