EUVIMEDOdkrijte EUVIMED
EvaluationsstudieOpen access

Evaluating Large Language Models for Standardizing Clinical Data Using Health Level Seven Fast Healthcare Interoperability Resources: Development and Evaluation Study

Álvaro Riquelme-Tornel, Pedro Luis Costa Del Amo, Catalina Martínez-Costa

JMIR Medical Informatics · 2026-11-13 · 14 · e100368-e100368

DOI: 10.2196/100368 ↗

At a glance

Evaluationsstudie in JMIR Medical Informatics, published online on 2026-11-13. Written by Álvaro Riquelme-Tornel and 2 co-authors. Institution: Department of Informatics and Systems, Faculty of Computer Science, Universidad de Murcia, Campus de Espinardo, Murcia, 30100, Spain. Major MEDLINE topics: Large Language Models, Health Information Interoperability, Health Level Seven. 53 references. The full text is freely available in PubMed Central.

PovzetekSubjectsBibliografski podatkiCiteRelated studies

Abstract (original language)

Background

One of the barriers to the wider adoption of clinical information modeling standards is the high cost associated with the time and effort required to model and map data from legacy systems, as well as the need for specialized modeling experts. Meanwhile, large language models (LLMs) are becoming powerful tools for data transformation and semantic processing.

Objective

The aim of this study was to evaluate the capability of LLMs to support the mapping of structured clinical data to Health Level Seven (HL7) Fast Healthcare Interoperability Resources (FHIR) resources and attributes.

Methods

Using data from the MIMIC-IV (Medical Information Mart for Intensive Care IV) database, a data pipeline integrating clustering techniques, retrieval-augmented resource grounding, and schema-guided prompting was implemented to support the FHIR mapping of structured clinical data. The pipeline was evaluated under 2 scenarios with different levels of complexity. In the first scenario, used as a baseline, the input data were well described and contextually rich. In the second scenario, designed as a controlled test under substantially reduced context, the original table structure and table-level descriptions were removed, while data element names and LLM-enriched descriptions were retained.

Results

Using 65 data elements, top 2 resource identification accuracy reached 100% in the baseline scenario. The highest mean attribute-level mapping accuracy was 80.61% (SD 2.06) for GPT-4o and 69.54% (SD 5.15) for Llama 3.3 70B. In the low-context scenario, clustering-based context recovery increased top 5 resource identification accuracy to 94%. Across temperature settings, GPT-4o achieved mean attribute-level mapping accuracies ranging from 73.23% (SD 2.92) to 76.77% (SD 1.14), whereas Llama 3.3 70B ranged from 54.31% (SD 4.17) to 58.31% (SD 2.23). Error analysis showed that the main remaining failures were associated with resource-level confusion and terminal-attribute semantic ambiguity, whereas other error categories were comparatively infrequent.

Conclusions

LLMs can support structured clinical data mapping to HL7 FHIR, but fine-grained semantic disambiguation remains the principal limitation. In context-rich settings, the main errors occurred when selecting the appropriate terminal FHIR attribute. Under reduced context, accurate candidate-resource retrieval became increasingly important, and clustering-based context recovery substantially improved resource identification. These findings support the use of LLMs as assistive components within human-supervised interoperability workflows, with particular attention to candidate-resource selection and ambiguity-sensitive attribute mappings.

Subjects

MeSH terms (MEDLINE)

Large Language ModelsHealth Information InteroperabilityHealth Level SevenHumansElectronic Health RecordsClustering Algorithms

Author keywords

Fast Healthcare Interoperability ResourcesHL7 FHIRHealth Level Sevenclinical data interoperabilityclusteringlarge language modelsretrieval-augmented generation

Authors and institutions

  • Department of Informatics and Systems, Faculty of Computer Science, Universidad de Murcia, Campus de Espinardo, Murcia, 30100, Spain.
  • Instituto Murciano de Investigación Biosanitaria, Murcia, Spain.
  • Latency Data, Madrid, Spain.

Cite

APA

Riquelme-Tornel, Á., Costa Del Amo, P. L., & Martínez-Costa, C. (2026). Evaluating Large Language Models for Standardizing Clinical Data Using Health Level Seven Fast Healthcare Interoperability Resources: Development and Evaluation Study. JMIR Medical Informatics, 14, e100368-e100368. https://doi.org/10.2196/100368

Vancouver

Riquelme-Tornel Á, Costa Del Amo PL, Martínez-Costa C. Evaluating Large Language Models for Standardizing Clinical Data Using Health Level Seven Fast Healthcare Interoperability Resources: Development and Evaluation Study. JMIR Medical Informatics. 2026;14:e100368-e100368. doi:10.2196/100368

BibTeX
@article{RiquelmeTornel2026,
  author = {Riquelme-Tornel, Álvaro and Costa Del Amo, Pedro Luis and Martínez-Costa, Catalina},
  title = {Evaluating Large Language Models for Standardizing Clinical Data Using Health Level Seven Fast Healthcare Interoperability Resources: Development and Evaluation Study},
  journal = {JMIR Medical Informatics},
  year = {2026},
  volume = {14},
  pages = {e100368--e100368},
  doi = {10.2196/100368}
}
Start for free →