Extracting International Classification of Diseases Codes from Clinical Documentation Using Large Language Models
Applied Clinical Informatics · 01.03.2025 · 16 (02) · 337-344
DOI: 10.1055/a-2491-3872 ↗Kurz gefasst
Fachartikel in der Zeitschrift Applied Clinical Informatics, erschienen am 01.03.2025. Verfasst von Ashley Simmons und 16 weiteren Autor:innen. Beteiligte Einrichtung: Department of Human Performance - Health Informatics and Information Management, West Virginia University, Morgantown, West Virginia, United States. Hauptthemen laut MEDLINE: International Classification of Diseases, Documentation, Natural Language Processing. Die Arbeit wurde bisher 20-mal zitiert (Crossref) und nennt 13 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: NIDDK NIH HHS, Digestive and Kidney Diseases NIH/NIDDK grant, U.S. Department of Health and Human Services.
Abstract (Originalsprache)
Background
Large language models (LLMs) have shown promise in various professional fields, including medicine and law. However, their performance in highly specialized tasks, such as extracting ICD-10-CM codes from patient notes, remains underexplored.
Objective
The primary objective was to evaluate and compare the performance of ICD-10-CM code extraction by different LLMs with that of human coder.
Methods
We evaluated performance of six LLMs (GPT-3.5, GPT-4, Claude 2.1, Claude 3, Gemini Advanced, and Llama 2-70b) in extracting ICD-10-CM codes against human coder. We used deidentified inpatient notes of authentic patient cases from American Health Information Management Association Vlab for this study. We calculated percent agreement and Cohen's kappa values to assess the agreement between LLMs and human coder. We then identified reasons for discrepancies in code extraction by LLMs in a 10% random subset.
Results
Among 50 inpatient notes, human coder extracted 165 unique ICD-10-CM codes. LLMs extracted significantly higher number of unique ICD-10-CM codes than human coder, with Llama 2-70b extracting most (658) and Gemini Advanced the least (221). GPT-4 achieved highest percent agreement with human coder at 15.2%, followed by Claude 3 (12.7%) and GPT-3.5 (12.4%). Cohen's kappa values indicated minimal to no agreement, ranging from −0.02 to 0.01. When focusing on primary diagnosis, Claude 3 achieved highest percent agreement (26%) and kappa value (0.25). Reasons for discrepancies in extraction of codes varied among LLMs and included extraction of codes for diagnoses not confirmed by providers (60% with GPT-4), extraction of nonspecific codes (25% with GPT-3.5), extraction of codes for signs and symptoms despite presence of more specific diagnosis (22% with Claude 2.1), and hallucinations (35% with Claude 2.1).
Conclusion
Current LLMs have poor performance in extraction of ICD-10-CM codes from inpatient notes when compared against the human coder.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Autor:innen und Institutionen
- Department of Human Performance - Health Informatics and Information Management, West Virginia University, Morgantown, West Virginia, United States.
- Division of Nephrology, Department of Medicine, Icahn School of Medicine at Mount Sinai, New York, New York, United States.
- Division of Nephrology, Department of Medicine, King Chulalongkorn Memorial Hospital, Chulalongkorn University, Bangkok, Thailand.
- Department of Emergency Medicine, West Virginia University, Morgantown, West Virginia, United States.
- Department of Orthopedics, West Virginia University, Morgantown, West Virginia, United States.
- Department of Radiology and Imaging Sciences, Emory University, Atlanta, Georgia, United States.
Förderung
- NIDDK NIH HHS
- Digestive and Kidney Diseases NIH/NIDDK grant
- U.S. Department of Health and Human Services
- National Institutes of Health
- National Institute of Diabetes
Zitieren
APA
Simmons, A., Takkavatakarn, K., McDougal, M., Dilcher, B., Pincavitch, J., Meadows, L., Kauffman, J., Klang, E., Wig, R., Smith, G., Soroush, A., Freeman, R., Apakama, D. J., Charney, A. W., Kohli-Seth, R., Nadkarni, G. N., & Sakhuja, A. (2025). Extracting International Classification of Diseases Codes from Clinical Documentation Using Large Language Models. Applied Clinical Informatics, 16(02), 337-344. https://doi.org/10.1055/a-2491-3872
Vancouver
Simmons A, Takkavatakarn K, McDougal M, Dilcher B, Pincavitch J, Meadows L, et al. Extracting International Classification of Diseases Codes from Clinical Documentation Using Large Language Models. Applied Clinical Informatics. 2025;16(02):337-344. doi:10.1055/a-2491-3872
BibTeX
@article{Simmons2025,
author = {Simmons, Ashley and Takkavatakarn, Kullaya and McDougal, Megan and Dilcher, Brian and Pincavitch, Jami and Meadows, Lukas and Kauffman, Justin and Klang, Eyal and Wig, Rebecca and Smith, Gordon and Soroush, Ali and Freeman, Robert and Apakama, Donald J and Charney, Alexander W and Kohli-Seth, Roopa and Nadkarni, Girish N and Sakhuja, Ankit},
title = {Extracting International Classification of Diseases Codes from Clinical Documentation Using Large Language Models},
journal = {Applied Clinical Informatics},
year = {2025},
volume = {16},
number = {02},
pages = {337--344},
doi = {10.1055/a-2491-3872}
}
EUVIMED