Harnessing large language models for reliable automatic Charlson Comorbidity Index scoring: a validation study
JAMIA Open · 09.09.2026 · 9 (5)
DOI: 10.1093/jamiaopen/ooag212 ↗Kurz gefasst
Fachartikel in der Zeitschrift JAMIA Open, online erschienen am 02.11.2026. Verfasst von Lana Mei Borcard und 6 weiteren Autor:innen. Beteiligte Einrichtung: Department of Orthopaedics, Balgrist University Hospital, University of Zurich, 8008 Zurich, Switzerland. Die Arbeit nennt 27 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Objectives
To determine whether large language models can compute the Charlson Comorbidity Index (CCI) from unstructured, German-language diagnosis lists with reliability approaching that of expert human consensus. Materials and Methods We retrospectively analyzed the complete free-text diagnosis lists of 200 patients from a tertiary orthopedic center, stratified by American Society of Anesthesiologists (ASA) physical status I-IV. Two board-certified orthopedic surgeons and 1 chiropractor independently assigned CCI values; a consensus reading served as the reference standard. Twelve open-source models, hosted on-premise, and 5 commercial models were each evaluated with and without guideline-based prompting. Agreement was quantified by accuracy, Cohen kappa, and the coefficient of determination, and discrepancies were classified using an adapted error taxonomy.
Results
Human interrater reliability was excellent (intraclass correlation 0.97). Across models, accuracy against consensus ranged from 0.34 to 0.79 and kappa from 0.27 to 0.75. The best open-source model (GPT-OSS-120B with guidelines) matched the best commercial model (coefficient of determination 0.93). Agreement declined with increasing ASA class, falling to 0.34-0.56 in ASA IV. Most discrepancies were knowledge-related (41.7%-58.3%); computation errors were rare, and 22.2%-32.0% arose from diagnosis lists too incomplete for any classification.
Discussion
Performance approached expert level in low-complexity records but degraded as documentation complexity increased, indicating that record quality rather than model reasoning is the dominant constraint. Locally hosted open-source models performed comparably to proprietary systems, allowing comorbidity extraction without transferring data off-premise.
Conclusion
Large language models can approximate expert CCI scoring directly from unstructured German-language clinical text, supporting scalable risk adjustment where documentation is adequate.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Orthopaedics, Balgrist University Hospital, University of Zurich, 8008 Zurich, Switzerland.
ORCID-Profile: Anna-Katharina Calek, Mazda Farshad, David Ephraim Bauer
Zitieren
APA
Borcard, L. M., Calek, A. K., Rebsamen, M., Herzog, B., Caprara, S., Farshad, M., & Bauer, D. E. (2026). Harnessing large language models for reliable automatic Charlson Comorbidity Index scoring: a validation study. JAMIA Open, 9(5). https://doi.org/10.1093/jamiaopen/ooag212
Vancouver
Borcard LM, Calek AK, Rebsamen M, Herzog B, Caprara S, Farshad M, et al. Harnessing large language models for reliable automatic Charlson Comorbidity Index scoring: a validation study. JAMIA Open. 2026;9(5). doi:10.1093/jamiaopen/ooag212
BibTeX
@article{Borcard2026,
author = {Borcard, Lana Mei and Calek, Anna-Katharina and Rebsamen, Michael and Herzog, Benedikt and Caprara, Sebastiano and Farshad, Mazda and Bauer, David Ephraim},
title = {Harnessing large language models for reliable automatic Charlson Comorbidity Index scoring: a validation study},
journal = {JAMIA Open},
year = {2026},
volume = {9},
number = {5},
doi = {10.1093/jamiaopen/ooag212}
}
EUVIMED