EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Development of a Framework for Evaluating Large Language Model Safety and Reliability: a Proof-of-Concept Evaluation

Fangyan Liu, Zhi Liu, Xiaolu Fei, Jingyu He, Jin Xing, Jia Li, Piu Chan

Journal of Medical Systems · 23.09.2026 · 50 (1)

DOI: 10.1007/s10916-026-02459-1 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Medical Systems, online erschienen am 23.09.2026. Verfasst von Fangyan Liu und 6 weiteren Autor:innen. Beteiligte Einrichtung: Department of Emergency, National Clinical Research Center for Geriatric Diseases, Xuanwu Hospital of Capital Medical University, Beijing, 100053, Ch…. Hauptthemen laut MEDLINE: Large Language Models, Decision Support Systems, Clinical. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 35 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Medical Science Research Project of Hebei, Science and Technology Innovation Program of Xiong'an New Area, National Science and Technology Major Project.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Large language models (LLMs) are entering clinical decision support faster than methodology can characterise their safety. Aggregate accuracy treats all errors as interchangeable and cannot support safe deployment under Software as a Medical Device (SaMD) and EU AI Act frameworks. To develop and demonstrate a framework for evaluating large language model safety and reliability using catastrophic-failure frequency and response reproducibility, incorporating a pre-specified error taxonomy, difficulty-stratified analysis, and two-layer response consistency. The framework was applied to 54 diagnostically challenging emergency cases from non-public institutional records. Six LLMs were accessed via application programming interfaces (APIs) (text-only, zero-shot, defaults; August–December 2025) and queried three times each, producing 972 physician-scored responses. Aggregate scores concealed heterogeneity: anchoring-bias failure (score ≤ 3) varied nearly four-fold (6.9%–26.4%) and rare-disease recognition six-fold (6.1%–37.9%); catastrophic (dangerous-recommendation, score ≤ 2) rates were two-tiered (1.5% for the safest two vs. 6.0% for the rest; p < 0.001), though within-tier ranks were inseparable (2–12 per model). Despite a middle-tier mean, GPT-5.1 was least reproducible (within-case SD 2.69) and fell within the higher catastrophic-failure tier; across models, 31 of 34 dangerous combinations were stochastic, not systematic—a failure mode hidden by aggregate metrics. Open-source DeepSeek R1 was statistically equivalent to Gemini 3 Pro within a 1.5-point margin by two one-sided tests (TOST: Δ − 0.09; 90% CI − 0.47 to 0.30; p < 0.001). In this single-centre proof-of-concept, aggregate accuracy was insufficient for safety characterisation: models with indistinguishable mean accuracy carried different catastrophic-failure tiers and reproducibility profiles. Error-type and response-consistency profiling may inform model selection, ensemble design, and conformity assessment.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Large Language ModelsDecision Support Systems, ClinicalHumansReproducibility of ResultsProof of Concept Study

Stichwörter der Autor:innen

AI governanceAI safety evaluationClinical decision support systemsHealth informaticsLarge language modelsResponse consistency

Autor:innen und Institutionen

  • Department of Emergency, National Clinical Research Center for Geriatric Diseases, Xuanwu Hospital of Capital Medical University, Beijing, 100053, China.
  • Department of Emergency, Xiongan Xuanwu Hospital, No. 1, Xili Road, Xiongan New Area, Hebei, 070001, China.
  • Information Center, National Clinical Research Center for Geriatric Diseases, Xuanwu Hospital of Capital Medical University, Beijing, 100053, China.
  • Information Center, Xiongan Xuanwu Hospital, No. 1, Xili Road, Xiongan New Area, Hebei, 070001, China.
  • Department of General Surgery, Xuanwu Hospital of Capital Medical University, No. 45, Changchun Street, Beijing, 100053, China. lijia@xwh.ccmu.edu.cn.
  • Department of General Surgery, Xiongan Xuanwu Hospital, No. 1, Xili Road, Xiongan New Area, Hebei, 070001, China. lijia@xwh.ccmu.edu.cn.

Förderung

  • Medical Science Research Project of Hebei
  • Science and Technology Innovation Program of Xiong'an New Area
  • National Science and Technology Major Project
  • Science and Technology Innovation Program of Xiong’an New Area

Zitieren

APA

Liu, F., Liu, Z., Fei, X., He, J., Xing, J., Li, J., & Chan, P. (2026). Development of a Framework for Evaluating Large Language Model Safety and Reliability: a Proof-of-Concept Evaluation. Journal of Medical Systems, 50(1). https://doi.org/10.1007/s10916-026-02459-1

Vancouver

Liu F, Liu Z, Fei X, He J, Xing J, Li J, et al. Development of a Framework for Evaluating Large Language Model Safety and Reliability: a Proof-of-Concept Evaluation. Journal of Medical Systems. 2026;50(1). doi:10.1007/s10916-026-02459-1

BibTeX
@article{Liu2026,
  author = {Liu, Fangyan and Liu, Zhi and Fei, Xiaolu and He, Jingyu and Xing, Jin and Li, Jia and Chan, Piu},
  title = {Development of a Framework for Evaluating Large Language Model Safety and Reliability: a Proof-of-Concept Evaluation},
  journal = {Journal of Medical Systems},
  year = {2026},
  volume = {50},
  number = {1},
  doi = {10.1007/s10916-026-02459-1}
}
Kostenlos starten →