EUVIMEDOpdag EUVIMED
Open access

Evaluating the Reasoning Capabilities of Large Language Models for Medical Coding and Hospital Readmission Risk Stratification: Zero-Shot Prompting Approach

Parvati Naliyatthaliyazchayil, Raajitha Muthyala, Judy Wawira Gichoya, Saptarshi Purkayastha

Journal of Medical Internet Research · 2025-07-30 · 27 · e74142-e74142

DOI: 10.2196/74142 ↗

At a glance

Journal article in Journal of Medical Internet Research, published online on 2025-07-30. Written by Parvati Naliyatthaliyazchayil and 3 co-authors. Institution: Department of Biomedical Engineering and Informatics, Luddy School of Informatics, Computing and Engineering, Indiana University Indianapolis, 535 W …. Major MEDLINE topics: Patient Readmission, Clinical Coding, Language, Decision Support Systems, Clinical. Cited 30 times so far (Crossref); 37 references. The full text is freely available in PubMed Central. Funded by: NHLBI NIH HHS, NIH HHS.

AbstractSubjectsBibliografiske oplysningerCiteRelated studies

Abstract (original language)

Background

Large language models (LLMs) such as ChatGPT-4, LLaMA-3.1, Gemini-1.5, DeepSeek-R1, and OpenAI-O3 have shown promising potential in health care, particularly for clinical reasoning and decision support. However, their reliability across critical tasks like diagnosis, medical coding, and risk prediction has received mixed reviews, especially in real-world settings without task-specific training.

Objective

This study aims to evaluate and compare the zero-shot performance of reasoning and nonreasoning LLMs in three essential clinical tasks: (1) primary diagnosis generation, (2) ICD-9 ( International Classification of Diseases, Ninth Revision ) medical code prediction, and (3) hospital readmission risk stratification. The goal is to assess whether these models can serve as general-purpose clinical decision support tools and to identify gaps in current capabilities.

Methods

Using the Medical Information Mart for Intensive Care-IV dataset, we selected a random cohort of 300 hospital discharge summaries. Prompts were engineered to include structured clinical content from 5 note sections: chief complaints, past medical history, surgical history, laboratories, and imaging. Prompts were standardized and zero-shot, with no model fine-tuning or repetition across runs. All model interactions were conducted through publicly available web user interfaces, without using application programming interfaces, to simulate real-world accessibility for nontechnical users. We incorporated rationale elicitation into prompts to evaluate model transparency, especially in reasoning models. Ground-truth labels were derived from the primary diagnosis documented in clinical notes, structured ICD-9 codes from diagnosis, and hospital-recorded readmission frequencies for risk stratification. Performance was measured using F 1 -scores and correctness percentages, and comparative performance was analyzed statistically.

Results

Among nonreasoning models, LLaMA-3.1 achieved the highest primary diagnosis accuracy (n=255, 85%), followed by ChatGPT-4 (n=254, 84.7%) and Gemini-1.5 (n=237, 79%). For ICD-9 prediction, correctness dropped significantly across all models: LLaMA-3.1 (n=128, 42.6%), ChatGPT-4 (n=122, 40.6%), and Gemini-1.5 (n=44, 14.6%). Hospital readmission risk prediction showed low performance in nonreasoning models: LLaMA-3.1 (n=124, 41.3%), Gemini-1.5 (n=122, 40.7%), and ChatGPT-4 (n=99, 33%). Among reasoning models, OpenAI-O3 outperformed in diagnosis (n=270, 90%) and ICD-9 coding (n=136, 45.3%), while DeepSeek-R1 performed slightly better in the readmission risk prediction (n=218, 72.6% vs O3’s n=212, 70.6%). Despite improved explainability, reasoning models generated verbose responses. None of the models met clinical standards across all tasks, and performance in medical coding remained the weakest area across all models.

Conclusions

Current LLMs exhibit moderate success in zero-shot diagnosis and risk prediction but underperform in ICD-9 code generation, reinforcing findings from prior studies. Reasoning models offer marginally better performance and increased interpretability, with limited reliability. Overall, statistical analysis between the models revealed that OpenAI-O3 outperformed the other models. These results highlight the need for task-specific fine-tuning and need human-in-the-loop checking. Future work will explore fine-tuning, stability through repeated trials, and evaluation on a different subset of deidentified real-world data with a larger sample size.

Subjects

MeSH terms (MEDLINE)

Patient ReadmissionClinical CodingLanguageDecision Support Systems, ClinicalHumansRisk AssessmentInternational Classification of DiseasesLarge Language Models

Author keywords

clinical decision supportexplainabilitylarge language modelsmedical codingprimary diagnosis predictionreadmission risk predictionzero-shot learning

Authors and institutions

  • Department of Biomedical Engineering and Informatics, Luddy School of Informatics, Computing and Engineering, Indiana University Indianapolis, 535 W Michigan Street, Indianapolis, IN, 46202, United States, 1 317 274 043…
  • Department of Radiology and Imaging Sciences, Emory University School of Medicine, Emory University, Atlanta, GA, United States.

Funding

  • NHLBI NIH HHS
  • NIH HHS

Cite

APA

Naliyatthaliyazchayil, P., Muthyala, R., Gichoya, J. W., & Purkayastha, S. (2025). Evaluating the Reasoning Capabilities of Large Language Models for Medical Coding and Hospital Readmission Risk Stratification: Zero-Shot Prompting Approach. Journal of Medical Internet Research, 27, e74142-e74142. https://doi.org/10.2196/74142

Vancouver

Naliyatthaliyazchayil P, Muthyala R, Gichoya JW, Purkayastha S. Evaluating the Reasoning Capabilities of Large Language Models for Medical Coding and Hospital Readmission Risk Stratification: Zero-Shot Prompting Approach. Journal of Medical Internet Research. 2025;27:e74142-e74142. doi:10.2196/74142

BibTeX
@article{Naliyatthaliyazchayil2025,
  author = {Naliyatthaliyazchayil, Parvati and Muthyala, Raajitha and Gichoya, Judy Wawira and Purkayastha, Saptarshi},
  title = {Evaluating the Reasoning Capabilities of Large Language Models for Medical Coding and Hospital Readmission Risk Stratification: Zero-Shot Prompting Approach},
  journal = {Journal of Medical Internet Research},
  year = {2025},
  volume = {27},
  pages = {e74142--e74142},
  doi = {10.2196/74142}
}
Start for free →