EUVIMEDOpdag EUVIMED
Open access

Developing an ICD-10 Coding Assistant: Pilot Study Using RoBERTa and GPT-4 for Term Extraction and Description-Based Code Selection

Sander Puts iD, Catharina M L Zegers iD, Andre Dekker iD, Iñigo Bermejo iD

JMIR Formative Research · 2025-02-11 · 9 · e60095-e60095

DOI: 10.2196/60095 ↗

At a glance

Journal article in JMIR Formative Research, published online on 2025-02-11. Written by Sander Puts and 3 co-authors. Institution: Department of Radiation Oncology (Maastro), GROW Research Institute for Oncology and Reproduction, Maastricht University Medical Centre+, P.O. Box 61…. Major MEDLINE topics: International Classification of Diseases, Clinical Coding, Artificial Intelligence, Natural Language Processing. Cited 18 times so far (Crossref); 34 references. The full text is freely available in PubMed Central.

AbstractSubjectsBibliografiske oplysningerCiteRelated studies

Abstract (original language)

Background

The International Classification of Diseases (ICD), developed by the World Health Organization, standardizes health condition coding to support health care policy, research, and billing, but artificial intelligence automation, while promising, still underperforms compared with human accuracy and lacks the explainability needed for adoption in medical settings.

Objective

The potential of large language models for assisting medical coders in the ICD-10 coding was explored through the development of a computer-assisted coding system. This study aimed to augment human coding by initially identifying lead terms and using retrieval-augmented generation (RAG)–based methods for computer-assisted coding enhancement.

Methods

The explainability dataset from the CodiEsp challenge (CodiEsp-X) was used, featuring 1000 Spanish clinical cases annotated with ICD-10 codes. A new dataset, CodiEsp-X-lead, was generated using GPT-4 to replace full-textual evidence annotations with lead term annotations. A Robustly Optimized BERT (Bidirectional Encoder Representations from Transformers) Pretraining Approach transformer model was fine-tuned for named entity recognition to extract lead terms. GPT-4 was subsequently employed to generate code descriptions from the extracted textual evidence. Using a RAG approach, ICD codes were assigned to the lead terms by querying a vector database of ICD code descriptions with OpenAI’s text-embedding-ada-002 model.

Results

The fine-tuned Robustly Optimized BERT Pretraining Approach achieved an overall F 1 -score of 0.80 for ICD lead term extraction on the new CodiEsp-X-lead dataset. GPT-4-generated code descriptions reduced retrieval failures in the RAG approach by approximately 5% for both diagnoses and procedures. However, the overall explainability F 1 -score for the CodiEsp-X task was limited to 0.305, significantly lower than the state-of-the-art F 1 -score of 0.633. The diminished performance was partly due to the reliance on code descriptions, as some ICD codes lacked descriptions, and the approach did not fully align with the medical coder’s workflow.

Conclusions

While lead term extraction showed promising results, the subsequent RAG-based code assignment using GPT-4 and code descriptions was less effective. Future research should focus on refining the approach to more closely mimic the medical coder’s workflow, potentially integrating the alphabetic index and official coding guidelines, rather than relying solely on code descriptions. This alignment may enhance system accuracy and better support medical coders in practice.

Subjects

MeSH terms (MEDLINE)

International Classification of DiseasesClinical CodingArtificial IntelligenceNatural Language ProcessingHumansPilot ProjectsSpain

Author keywords

AI automationBidirectional Encoder Representations from TransformersGPT-4ICD-10International Classification of DiseasesLLMNERRAGRoBERTaRobustly Optimized BERT Pretraining Approachartificial intelligencecode analysiscodingcomputer assisted codingcomputer-assisted-coding

Authors and institutions

  • Department of Radiation Oncology (Maastro), GROW Research Institute for Oncology and Reproduction, Maastricht University Medical Centre+, P.O. Box 616, Maastricht, 6200 MD, Netherlands, 31 43 38 81863.
  • Data Science Institute (DSI), Hasselt University, Belgium.

ORCID profiles: Sander Puts, Catharina M L Zegers, Andre Dekker, Iñigo Bermejo

Cite

APA

Puts, S., Zegers, C. M. L., Dekker, A., & Bermejo, I. (2025). Developing an ICD-10 Coding Assistant: Pilot Study Using RoBERTa and GPT-4 for Term Extraction and Description-Based Code Selection. JMIR Formative Research, 9, e60095-e60095. https://doi.org/10.2196/60095

Vancouver

Puts S, Zegers CML, Dekker A, Bermejo I. Developing an ICD-10 Coding Assistant: Pilot Study Using RoBERTa and GPT-4 for Term Extraction and Description-Based Code Selection. JMIR Formative Research. 2025;9:e60095-e60095. doi:10.2196/60095

BibTeX
@article{Puts2025,
  author = {Puts, Sander and Zegers, Catharina M L and Dekker, Andre and Bermejo, Iñigo},
  title = {Developing an ICD-10 Coding Assistant: Pilot Study Using RoBERTa and GPT-4 for Term Extraction and Description-Based Code Selection},
  journal = {JMIR Formative Research},
  year = {2025},
  volume = {9},
  pages = {e60095--e60095},
  doi = {10.2196/60095}
}
Start for free →