EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Harnessing large language models for reliable automatic Charlson Comorbidity Index scoring: a validation study

Lana Mei Borcard, Anna-Katharina Calek iD, Michael Rebsamen, Benedikt Herzog, Sebastiano Caprara, Mazda Farshad iD, David Ephraim Bauer iD

JAMIA Open · 09.09.2026 · 9 (5)

DOI: 10.1093/jamiaopen/ooag212 ↗

Kurz gefasst

Fachartikel in der Zeitschrift JAMIA Open, online erschienen am 02.11.2026. Verfasst von Lana Mei Borcard und 6 weiteren Autor:innen. Beteiligte Einrichtung: Department of Orthopaedics, Balgrist University Hospital, University of Zurich, 8008 Zurich, Switzerland. Die Arbeit nennt 27 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objectives

To determine whether large language models can compute the Charlson Comorbidity Index (CCI) from unstructured, German-language diagnosis lists with reliability approaching that of expert human consensus. Materials and Methods We retrospectively analyzed the complete free-text diagnosis lists of 200 patients from a tertiary orthopedic center, stratified by American Society of Anesthesiologists (ASA) physical status I-IV. Two board-certified orthopedic surgeons and 1 chiropractor independently assigned CCI values; a consensus reading served as the reference standard. Twelve open-source models, hosted on-premise, and 5 commercial models were each evaluated with and without guideline-based prompting. Agreement was quantified by accuracy, Cohen kappa, and the coefficient of determination, and discrepancies were classified using an adapted error taxonomy.

Results

Human interrater reliability was excellent (intraclass correlation 0.97). Across models, accuracy against consensus ranged from 0.34 to 0.79 and kappa from 0.27 to 0.75. The best open-source model (GPT-OSS-120B with guidelines) matched the best commercial model (coefficient of determination 0.93). Agreement declined with increasing ASA class, falling to 0.34-0.56 in ASA IV. Most discrepancies were knowledge-related (41.7%-58.3%); computation errors were rare, and 22.2%-32.0% arose from diagnosis lists too incomplete for any classification.

Discussion

Performance approached expert level in low-complexity records but degraded as documentation complexity increased, indicating that record quality rather than model reasoning is the dominant constraint. Locally hosted open-source models performed comparably to proprietary systems, allowing comorbidity extraction without transferring data off-premise.

Conclusion

Large language models can approximate expert CCI scoring directly from unstructured German-language clinical text, supporting scalable risk adjustment where documentation is adequate.

Schlagwörter

Stichwörter der Autor:innen

Charlson Comorbidity Indexcomorbidity extractionelectronic health recordslarge language modelsnatural language processing

Autor:innen und Institutionen

  • Department of Orthopaedics, Balgrist University Hospital, University of Zurich, 8008 Zurich, Switzerland.

ORCID-Profile: Anna-Katharina Calek, Mazda Farshad, David Ephraim Bauer

Zitieren

APA

Borcard, L. M., Calek, A. K., Rebsamen, M., Herzog, B., Caprara, S., Farshad, M., & Bauer, D. E. (2026). Harnessing large language models for reliable automatic Charlson Comorbidity Index scoring: a validation study. JAMIA Open, 9(5). https://doi.org/10.1093/jamiaopen/ooag212

Vancouver

Borcard LM, Calek AK, Rebsamen M, Herzog B, Caprara S, Farshad M, et al. Harnessing large language models for reliable automatic Charlson Comorbidity Index scoring: a validation study. JAMIA Open. 2026;9(5). doi:10.1093/jamiaopen/ooag212

BibTeX
@article{Borcard2026,
  author = {Borcard, Lana Mei and Calek, Anna-Katharina and Rebsamen, Michael and Herzog, Benedikt and Caprara, Sebastiano and Farshad, Mazda and Bauer, David Ephraim},
  title = {Harnessing large language models for reliable automatic Charlson Comorbidity Index scoring: a validation study},
  journal = {JAMIA Open},
  year = {2026},
  volume = {9},
  number = {5},
  doi = {10.1093/jamiaopen/ooag212}
}
Kostenlos starten →