BEYOND MODEL SCALE: A TASK-AWARE RELIABILITY BENCHMARK FOR LARGE LANGUAGE MODELS IN CLINICAL DECISION SUPPORT
Innovation and Technology Studies · 2024
DOI: 10.61784/its4026 ↗Kurz gefasst
Fachartikel in der Zeitschrift Innovation and Technology Studies, 2024. Verfasst von JiTong Zou und 2 weiteren Autor:innen. Die Arbeit wurde bisher 3-mal zitiert (Crossref). Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Model size is often treated as a proxy for trustworthiness when large language models (LLMs) are considered for clinical decision support, yet a single aggregate score can hide task-specific failure modes. This paper introduces TARB-CDS, a task-aware reliability audit that transforms a model-by-task accuracy matrix into deployment-relevant reliability profiles. The framework combines macro accuracy with lower-tail conditional value at risk over the two weakest tasks (CVaR-2), worst-task accuracy, Pareto dominance, policy-weighted scores, contextual rank stability under uncertain task mixtures, output-format reliability, and matched base-versus-instruction-tuned comparisons. We conduct a reproducible secondary analysis of 12 LLM checkpoints evaluated on seven Med-HALT reasoning and biomedical-memory tasks. The public reasoning files contain 39,590 rows across False Confidence, Fake Question, and None-of-the-Above tests; published model accuracies and format-error rates are analyzed without claiming fresh model inference. Falcon-40B has the highest seven-task macro accuracy (42.67%) and CVaR-2 (9.36%), but the best worst-task accuracy among all models is only 1.38%. Under 100,000 uniformly sampled task mixtures, Falcon-40B ranks first in 77.60% of contexts, leaving substantial rank instability. Among open checkpoints, parameter count has only a modest descriptive association with macro accuracy (Spearman ρ=0.366), while all five matched instruction/chat-tuning pairs reduce macro accuracy by an average of 9.30 percentage points. These results show that scale alone is insufficient for reliability claims and motivate task-conditional model cards and minimum-performance gates before clinical evaluation.
Zitieren
APA
Zou, J., Qin, Y., & Rhee, M. (2024). BEYOND MODEL SCALE: A TASK-AWARE RELIABILITY BENCHMARK FOR LARGE LANGUAGE MODELS IN CLINICAL DECISION SUPPORT. Innovation and Technology Studies. https://doi.org/10.61784/its4026
Vancouver
Zou J, Qin Y, Rhee M. BEYOND MODEL SCALE: A TASK-AWARE RELIABILITY BENCHMARK FOR LARGE LANGUAGE MODELS IN CLINICAL DECISION SUPPORT. Innovation and Technology Studies. 2024. doi:10.61784/its4026
BibTeX
@article{Zou2024,
author = {Zou, JiTong and Qin, YuXuan and Rhee, MinJae},
title = {BEYOND MODEL SCALE: A TASK-AWARE RELIABILITY BENCHMARK FOR LARGE LANGUAGE MODELS IN CLINICAL DECISION SUPPORT},
journal = {Innovation and Technology Studies},
year = {2024},
doi = {10.61784/its4026}
}
EUVIMED