Validation of 13 102 International Classification of Diseases, Tenth Revision, Clinical Modification codes using a large language model-based system
Journal of the American Medical Informatics Association · 2026
DOI: 10.1093/jamia/ocag008 ↗Kurz gefasst
Validierungsstudie in der Zeitschrift Journal of the American Medical Informatics Association, 2026. Verfasst von Yichen Wang und 13 weiteren Autor:innen. Beteiligte Einrichtung: Division of Gastroenterology and Hepatology, Department of Medicine, Mayo Clinic, Jacksonville, FL 32224, United States. Hauptthemen laut MEDLINE: Large Language Models, International Classification of Diseases, Clinical Coding. Die Arbeit wurde bisher 4-mal zitiert (Crossref) und nennt 21 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: Mayo Clinic Center for Digital Health AI/ML Enablement, Dalio Philanthropies.
Abstract (Originalsprache)
Objectives
To comprehensively evaluate the validity of International Classification of Diseases, Tenth Revision, Clinical Modification (ICD-10-CM) codes for both prevalent diagnoses and less common diseases, and to assess the performance of a large language model (LLM)-based system in validating these codes. Materials and Methods This retrospective study analyzed hospital admissions from the Medical Information Mart for Intensive Care IV (MIMIC-IV) database. We developed a validated LLM-based system using GPT-4o, refined through iterative prompt engineering, to assess ICD-10-CM code validity. We measured the positive predictive value (PPV) of ICD-10-CM codes, PPV of principal and secondary diagnoses, and the performance of an LLM-based system in code validation.
Results
Among 865 079 assigned codes, the PPV was 84.6% (95% CI, 84.5%-84.6%). Principal diagnoses had a PPV of 93.9% (95% CI, 93.7%-94.1%), while secondary diagnoses had a PPV of 83.8% (95% CI, 83.7%-83.9%). The LLM system demonstrated high performance in validating ICD codes, achieving 93.6% accuracy, 95.4% sensitivity, and 85.2% specificity. Among correctly assigned secondary diagnoses, the majority (67.9%) represented historical or baseline conditions, while 32.1% reflected active conditions that deviated from baseline status; 22.3% of these emerged after hospital admission. PPV decreases with later diagnosis positions, with the largest decline occurring between principal and secondary diagnoses. Discussion and Conclusion In this large-scale evaluation, ICD-10-CM codes exhibited generally high accuracy, though variability existed by position and condition type. A validated LLM system performed comparably to physician review and offers a scalable means to improve coding accuracy. These findings support the potential for integrating LLM-based auditing into routine workflows to strengthen the quality of administrative and research data.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Division of Gastroenterology and Hepatology, Department of Medicine, Mayo Clinic, Jacksonville, FL 32224, United States.
- Division of Hospital Medicine, Department of Medicine, Perelman School of Medicine at the University of Pennsylvania, Philadelphia, PA 19104, United States.
- Department of Medicine, University of Maryland Medical Center Midtown Campus, Baltimore, MD 21201, United States.
- Department of Medicine, Mayo Clinic, Jacksonville, FL 32224, United States.
- Department of Computing, Xi'an Jiaotong-Liverpool University, Suzhou, Jiangsu 215123, China.
- Ponte Vedra High School, Ponte Vedra, FL 32081, United States.
ORCID-Profile: Yichen Wang, Yuting Huang
Förderung
- Mayo Clinic Center for Digital Health AI/ML Enablement
- Dalio Philanthropies
Zitieren
APA
Wang, Y., Song, Y., Siu, R., Nimma, I. R., Yan, Y., Savage, T. R., Wang, Y., Li, Z., Ramai, D., Wang, J., Badurdeen, D., Tao, C., Kumbhari, V., & Huang, Y. (2026). Validation of 13 102 International Classification of Diseases, Tenth Revision, Clinical Modification codes using a large language model-based system. Journal of the American Medical Informatics Association. https://doi.org/10.1093/jamia/ocag008
Vancouver
Wang Y, Song Y, Siu R, Nimma IR, Yan Y, Savage TR, et al. Validation of 13 102 International Classification of Diseases, Tenth Revision, Clinical Modification codes using a large language model-based system. Journal of the American Medical Informatics Association. 2026. doi:10.1093/jamia/ocag008
BibTeX
@article{Wang2026,
author = {Wang, Yichen and Song, Yilin and Siu, Rex and Nimma, Induja R and Yan, Yan and Savage, Thomas R and Wang, Yiming and Li, Zhichen and Ramai, Daryl and Wang, Jiale and Badurdeen, Dilhana and Tao, Cui and Kumbhari, Vivek and Huang, Yuting},
title = {Validation of 13 102 International Classification of Diseases, Tenth Revision, Clinical Modification codes using a large language model-based system},
journal = {Journal of the American Medical Informatics Association},
year = {2026},
doi = {10.1093/jamia/ocag008}
}
EUVIMED