EUVIMEDOdkrijte EUVIMED
ValidierungsstudieOpen access

Predicting 30-Day Postoperative Mortality and American Society of Anesthesiologists Physical Status Using Retrieval-Augmented Large Language Models: Development and Validation Study

Ying-Hao Chen iD, Shanq-Jang Ruan iD, Pei-Fu Chen iD

Journal of Medical Internet Research · 2025

DOI: 10.2196/75052 ↗

At a glance

Validierungsstudie in Journal of Medical Internet Research, 2025. Written by Ying-Hao Chen and 2 co-authors. Institution: Department of Electronic and Computer Engineering, National Taiwan University of Science and Technology, Taipei, Taiwan. Major MEDLINE topics: Postoperative Complications. Cited 14 times so far (Crossref); 46 references. The full text is freely available in PubMed Central.

PovzetekSubjectsBibliografski podatkiCiteRelated studies

Abstract (original language)

Background

Accurately assessing perioperative risk is critical for informed surgical planning and patient safety. However, current prediction models often rely on structured data and overlook the nuanced clinical reasoning embedded in free-text preoperative notes. Recent advances in large language models (LLMs) have opened opportunities for harnessing unstructured clinical data, yet their application in perioperative prediction remains limited by concerns about factual accuracy. Retrieval-augmented generation (RAG) offers a promising solution—enhancing LLM performance by grounding outputs in domain-specific knowledge sources, potentially improving both predictive accuracy and clinical interpretability.

Objective

This study aimed to investigate whether integrating LLMs with RAG can improve the prediction of 30-day postoperative mortality and American Society of Anesthesiologists (ASA) physical status classification using unstructured preoperative clinical notes.

Methods

We conducted a retrospective cohort study using 24,491 medical records from a tertiary medical center, including preoperative anesthesia assessments, discharge summaries, and surgical information. To extract clinical insights from free-text data, we used the LLaMA 3.1-8B language model with RAG, using MedEmbed for text embedding and Miller’s Anesthesia as the primary retrieval source. We evaluated model performance under various configurations, including embedding models, chunk sizes, and few-shot prompting. Machine learning (ML) models, including random forest, support vector machines (SVM), Extreme Gradient Boosting (XGBoost), and logistic regression, were trained on structured features as baselines.

Results

A total of 520 (2.1%) patients experienced in-hospital 30-day postoperative mortality. The ASA physical status distribution was as follows: class I: 535 (2.2%); class II: 15,272 (62.4%); class III: 8024 (32.8%); class IV: 606 (2.5%); and class V: 54 (0.22%). For 30-day postoperative mortality prediction, the LLaMA‑RAG model achieved an F1-score of 0.4663 (95% CI 0.4654-0.4672), versus 0.2369 (95% CI 0.2341-0.2397) without few‑shot prompting, 0.0879 (95% CI 0.0717-0.1041) without RAG, and 0.0436 (95% CI 0.0292-0.0580) without either few‑shot prompting or RAG. Among ML models, XGBoost scored 0.4459 (95% CI 0.4176-0.4742); random forest, 0.3953 (95% CI 0.3791-0.4115); logistic regression, 0.2720 (95% CI 0.2647-0.2793); and SVM, 0.2474 (95% CI 0.2275-0.2673). For ASA classification, LLaMA‑RAG achieved a micro F1-score of 0.8409 (95% CI 0.8238-0.8551) versus 0.6546 (95% CI 0.6430-0.6796) without few-shot prompting, 0.6340 (95% CI 0.6157-0.6535) without RAG, and 0.4238 (95% CI 0.3952-0.4490) without either few‑shot prompting or RAG. In comparison, XGBoost achieved 0.8273 (95% CI 0.8209-0.8498); logistic regression, 0.7940 (95% CI 0.7671-0.7950); random forest, 0.7847 (95% CI 0.7637-0.7868); and SVM, 0.7697 (95% CI 0.7637-0.7697). Notably, the model demonstrated exceptional sensitivity in identifying rare but high-risk cases, such as ASA Class 5 patients and postoperative deaths.

Conclusions

The LLaMA-RAG model significantly improved the prediction of postoperative mortality and ASA classification, especially for rare high-risk cases. By grounding outputs in domain knowledge, retrieval-augmented generation enhanced both accuracy and prompt‑driven interpretability over ML and ablation models—highlighting its promise for real-world clinical decision support.

Subjects

MeSH terms (MEDLINE)

Postoperative ComplicationsHumansRetrospective StudiesAnesthesiologistsMaleFemaleAgedMiddle AgedSocieties, MedicalPostoperative PeriodUnited StatesLanguageLarge Language Models

Author keywords

few-shot promptingmachine learningperioperative careprediction modelsurgical risk stratificationunstructured clinical data

Authors and institutions

  • Department of Electronic and Computer Engineering, National Taiwan University of Science and Technology, Taipei, Taiwan.
  • Department of Anesthesiology, Far Eastern Memorial Hospital, New Taipei City, Taiwan.
  • Department of Electrical Engineering, Yuan Ze University, Taoyuan, Taiwan.

ORCID profiles: Ying-Hao Chen, Shanq-Jang Ruan, Pei-Fu Chen

Cite

APA

Chen, Y. H., Ruan, S. J., & Chen, P. F. (2025). Predicting 30-Day Postoperative Mortality and American Society of Anesthesiologists Physical Status Using Retrieval-Augmented Large Language Models: Development and Validation Study. Journal of Medical Internet Research. https://doi.org/10.2196/75052

Vancouver

Chen YH, Ruan SJ, Chen PF. Predicting 30-Day Postoperative Mortality and American Society of Anesthesiologists Physical Status Using Retrieval-Augmented Large Language Models: Development and Validation Study. Journal of Medical Internet Research. 2025. doi:10.2196/75052

BibTeX
@article{Chen2025,
  author = {Chen, Ying-Hao and Ruan, Shanq-Jang and Chen, Pei-Fu},
  title = {Predicting 30-Day Postoperative Mortality and American Society of Anesthesiologists Physical Status Using Retrieval-Augmented Large Language Models: Development and Validation Study},
  journal = {Journal of Medical Internet Research},
  year = {2025},
  doi = {10.2196/75052}
}
Start for free →