American Society of Anesthesiologists Physical Status Classification: Hybrid Machine Learning–Large Language Model Ensemble Retrospective Validation Study
Journal of Medical Internet Research · 2026-11-05 · 28 · e90724-e90724
DOI: 10.2196/90724 ↗At a glance
Validierungsstudie in Journal of Medical Internet Research, published online on 2026-11-05. Written by Christabelle Pabalan and 3 co-authors. Institution: Qventus Inc, Mountain View, CA, United States. Major MEDLINE topics: Machine Learning, Anesthesiologists. 39 references. The full text is freely available in PubMed Central.
Abstract (original language)
Background
The American Society of Anesthesiologists Physical Status (ASA-PS) classification system fundamentally shapes perioperative care delivery but suffers from poor interrater reliability (0.4‐0.6). Machine learning (ML) models and large language models (LLMs) offer complementary remedies: ML processes structured data consistently but cannot interpret the nuance of clinical narratives, while LLMs parse narrative text and generate explanations but adhere inconsistently to diagnostic criteria.
Objective
This study aimed to develop and evaluate a parallel ML-LLM ensemble that combines the complementary strengths of both approaches for automated ASA-PS classification.
Methods
We retrospectively analyzed 3000 adult surgical encounters from the University of Arkansas for Medical Sciences (UAMS) between August 2024 and May 2025. Cases were randomly allocated to training (n=2000), validation (N=500), and held-out test sets (N=500). The validation set was used exclusively for architecture comparison and hyperparameter tuning, and the held-out test set was reserved for final model performance validation. We developed multiple architectures, including traditional ML models (Extreme Gradient Boosting [XGBoost], Light Gradient Boosting Machine [LightGBM], ExtraTrees), an LLM-only baseline (Generative Pretrained Transformer-4o [GPT-4o]), and hybrid approaches. The parallel ensemble processed structured data through XGBoost and unstructured clinical notes through GPT-4o independently, with outputs combined via weighted averaging. Model performance was evaluated using macro- F 1 -score, exact match accuracy, and ±1-class accuracy.
Results
A theoretical performance ceiling of macro- F 1 -score=0.59 was determined through evaluation by an expert panel of 2 board-certified anesthesiologists (AF and GF) who independently rated 50 patient charts for ASA-PS scores. On the true held-out test set (N=500), the parallel ensemble ( α =0.30) achieved a macro- F 1 -score of 0.530 (95% CI 0.350‐0.600), exact match accuracy of 0.630 (95% CI 0.580‐0.670), and ±1-class accuracy of 0.980 (95% CI 0.968‐0.992), exceeding the frozen LLM-only baseline evaluated on the same set (macro- F 1 -score 0.493, exact match accuracy 0.621,±1-class accuracy 0.981). On the validation set used for architecture selection, the ensemble outperformed structured-only ML (XGBoost macro- F 1 -score=0.333, P
Subjects
MeSH terms (MEDLINE)
Author keywords
Authors and institutions
- Qventus Inc, Mountain View, CA, United States.
- Department of Anesthesiology, University of Arkansas Medical Center, Little Rock, AR, United States.
- Department Anesthesiology and Perioperative Medicine, Medical University of South Carolina, 135 Rutledge Ave, Charleston, SC, 29425, United States, +1 (843) 792-2322.
Cite
APA
Pabalan, C., Stewart, A., Fisher, G., & Fisher, A. (2026). American Society of Anesthesiologists Physical Status Classification: Hybrid Machine Learning–Large Language Model Ensemble Retrospective Validation Study. Journal of Medical Internet Research, 28, e90724-e90724. https://doi.org/10.2196/90724
Vancouver
Pabalan C, Stewart A, Fisher G, Fisher A. American Society of Anesthesiologists Physical Status Classification: Hybrid Machine Learning–Large Language Model Ensemble Retrospective Validation Study. Journal of Medical Internet Research. 2026;28:e90724-e90724. doi:10.2196/90724
BibTeX
@article{Pabalan2026,
author = {Pabalan, Christabelle and Stewart, Alvin and Fisher, Gabrielle and Fisher, Andrew},
title = {American Society of Anesthesiologists Physical Status Classification: Hybrid Machine Learning–Large Language Model Ensemble Retrospective Validation Study},
journal = {Journal of Medical Internet Research},
year = {2026},
volume = {28},
pages = {e90724--e90724},
doi = {10.2196/90724}
}
EUVIMED