EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Evaluation of Small and Large Language Models for Calculation of the ASA Score and Charlson Comorbidity Index in Orthopedic Surgical Patients: A Retrospective Concordance Analysis

Marco Di Maio, Giorgio Stopper, Vincenzo Di Matteo, Katia Chiappetta, Guido Grappiolo, Mattia Loppini iD

Bioengineering · 19.08.2026 · 13 (8) · 936

DOI: 10.3390/bioengineering13080936 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Bioengineering, online erschienen am 19.08.2026. Verfasst von Marco Di Maio und 5 weiteren Autor:innen. Beteiligte Einrichtung: Department of Biomedical Sciences, Humanitas University, Via Rita Levi Montalcini 4, Pieve Emanuele, 20072 Milan, Italy. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 35 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

The ASA Physical Status (ASA-PS) classification and the Charlson Comorbidity Index (CCI) are common pre-operative scoring tools. Language models could automate structured pre-operative scoring, but direct comparisons require paired inference because all models are evaluated on the same patients.

Methods

In this retrospective single-center concordance analysis, 101 consecutive adult orthopedic patients were independently rated by two clinicians; the rounded mean for ASA-PS and arithmetic mean for CCI formed a clinician-derived composite reference. The cohort contained no ASA-PS IV-V patients. Six model configurations received identical prompts. Agreement was assessed using quadratic weighted kappa, ICC(2,1), exact and adjacent agreement, MAD, RMSE, and Bland–Altman limits. Post hoc between-model comparisons used 10,000 patient-level paired bootstrap replicates with Benjamini–Hochberg correction.

Results

Inter-clinician weighted kappa was 0.713 for ASA-PS and 0.914 for CCI. GPT-5.2 reached kappa 0.884 for ASA-PS and 0.970 for CCI. In paired analyses, GPT-5.2 had significantly higher quadratic weighted kappa than every other tested model for both outcomes and significantly higher ICC for CCI after multiplicity correction. Phi4 and deepseek-r1-70B were not significantly different from inter-clinician agreement for CCI kappa or ICC; equivalence was not tested.

Conclusions

Among the six evaluated model configurations, GPT-5.2 achieved significantly higher agreement with the clinician-derived composite reference than the other tested models for both ASA-PS and CCI in post hoc paired analyses with multiplicity correction. Locally deployable phi4 and deepseek-r1-70B showed CCI agreement estimates that were not statistically distinguishable from inter-clinician agreement, although equivalence was not tested. These findings are limited to the evaluated models and study cohort.

Schlagwörter

Stichwörter der Autor:innen

ASA physical statusCharlson comorbidity indexclinical natural language processinghealth data privacyinter-rater agreementlarge language modelson-premise deploymentorthopedic surgerypre-operative risk stratificationsmall language models

Autor:innen und Institutionen

  • Department of Biomedical Sciences, Humanitas University, Via Rita Levi Montalcini 4, Pieve Emanuele, 20072 Milan, Italy.
  • IRCCS Humanitas Research Hospital, Via Manzoni 56, Rozzano, 20089 Milan, Italy.
  • School of Medicine, Degree Course in Medicine and Surgery, Humanitas University, Via Rita Levi Montalcini 4, Pieve Emanuele, 20072 Milan, Italy.

ORCID-Profile: Mattia Loppini

Zitieren

APA

Di Maio, M., Stopper, G., Di Matteo, V., Chiappetta, K., Grappiolo, G., & Loppini, M. (2026). Evaluation of Small and Large Language Models for Calculation of the ASA Score and Charlson Comorbidity Index in Orthopedic Surgical Patients: A Retrospective Concordance Analysis. Bioengineering, 13(8), 936. https://doi.org/10.3390/bioengineering13080936

Vancouver

Di Maio M, Stopper G, Di Matteo V, Chiappetta K, Grappiolo G, Loppini M. Evaluation of Small and Large Language Models for Calculation of the ASA Score and Charlson Comorbidity Index in Orthopedic Surgical Patients: A Retrospective Concordance Analysis. Bioengineering. 2026;13(8):936. doi:10.3390/bioengineering13080936

BibTeX
@article{DiMaio2026,
  author = {Di Maio, Marco and Stopper, Giorgio and Di Matteo, Vincenzo and Chiappetta, Katia and Grappiolo, Guido and Loppini, Mattia},
  title = {Evaluation of Small and Large Language Models for Calculation of the ASA Score and Charlson Comorbidity Index in Orthopedic Surgical Patients: A Retrospective Concordance Analysis},
  journal = {Bioengineering},
  year = {2026},
  volume = {13},
  number = {8},
  pages = {936},
  doi = {10.3390/bioengineering13080936}
}
Kostenlos starten →