EUVIMEDEUVIMED entdecken
VergleichsstudieOpen Access

Using a Diverse Test Suite to Assess Large Language Models on Fast Health Care Interoperability Resources Knowledge: Comparative Analysis

Ahmad Idrissi-Yaghir, Kamyar Arzideh, Henning Schäfer, Bahadir Eryilmaz, Mikel Bahn, Yutong Wen, Katarzyna Borys, Eva Hartmann, Cynthia Schmidt, Obioma Pelka, Johannes Haubold, Christoph M Friedrich und 2 weitere

Journal of Medical Internet Research · 12.08.2025 · 27 · e73540-e73540

DOI: 10.2196/73540 ↗

Kurz gefasst

Vergleichsstudie in der Zeitschrift Journal of Medical Internet Research, online erschienen am 12.08.2025. Verfasst von Ahmad Idrissi-Yaghir und 13 weiteren Autor:innen. Beteiligte Einrichtung: Institute of Diagnostic and Interventional Radiology and Neuroradiology, University Hospital Essen, Essen, Germany. Hauptthemen laut MEDLINE: Natural Language Processing, Health Information Interoperability. Die Arbeit wurde bisher 5-mal zitiert (Crossref) und nennt 28 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Recent natural language processing breakthroughs, particularly with the emergence of large language models (LLMs), have demonstrated remarkable capabilities on general knowledge benchmarks. However, there is limited data on the performance and understanding of these models in relation to the Fast Healthcare Interoperability Resources (FHIR) standard. The complexity and specialized nature of FHIR present challenges for LLMs, which are typically trained on broad datasets and may have a limited understanding of the nuances required for domain-specific tasks. Improving health data interoperability can greatly benefit the use of clinical data and interaction with electronic health records.

Objective

This study presents the Fast Healthcare Interoperability Resources (FHIR) Workbench, a comprehensive suite of datasets designed to evaluate the ability of LLMs to understand and apply the FHIR standard.

Methods

In total, 4 evaluation datasets were created to assess the FHIR knowledge and capabilities of LLMs. These tasks include multiple-choice questions on general FHIR concepts and the FHIR Representational State Transfer (REST) application programming interface, as well as correctly identifying the resource type and generating FHIR resources from unstructured clinical patient notes. In addition, we evaluate open-source LLMs, such as Qwen 2.5 Coder and DeepSeek-V3, and commercial LLMs, including GPT-4o and Gemini 2, on these tasks in a zero-shot setting. To provide context for interpreting LLM performance, a subset of the datasets was human-evaluated by recruiting 6 participants with varying levels of FHIR expertise.

Results

Our evaluation across multiple FHIR tasks revealed nuanced performance metrics. Commercial models demonstrated exceptional capabilities, with GPT-4o achieving a 0.9990 F 1 -score on the FHIR-ResourceID task, 0.9400 on the FHIR-QA task, and 0.9267 on the FHIR-RESTQA task. Open-source models also demonstrated strong performance, with DeepSeek-v3 achieving 0.9400 on FHIR-QA, 0.9400 on FHIR-RESTQA, and 0.9142 on FHIR-ResourceID. Qwen 2.5 Coder-7B-Instruct demonstrated high accuracy, scoring 0.9533 on FHIR-QA and 0.8920 on FHIR-ResourceID. However, all models struggled with the Note2FHIR task, with performance ranging from 0.0382 (OLMo) to a maximum of 0.3633 (GPT-4.5-preview), highlighting the significant challenge of converting unstructured clinical text into FHIR-compliant resources. Human participants achieved accuracy scores ranging from 0.50 to 1.0 across the first 3 tasks.

Conclusions

This study highlights the competitive performance of both open-source models, such as Qwen and DeepSeek, and commercial models, such as GPT-4o and Gemini, in FHIR-related tasks. While open-source models are advancing rapidly, commercial models still have an advantage for specific, complex tasks. The FHIR Workbench offers a valuable platform for evaluating the capabilities of these models and promoting improvements in health data interoperability.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Natural Language ProcessingHealth Information InteroperabilityHumansElectronic Health RecordsLarge Language Models

Stichwörter der Autor:innen

DeepSeekGPT-4LLMshealth data interoperabilitylarge language modelsmedical informatics

Autor:innen und Institutionen

  • Institute of Diagnostic and Interventional Radiology and Neuroradiology, University Hospital Essen, Essen, Germany.
  • Institute for Artificial Intelligence in Medicine, University Hospital Essen, Hufelandstraße 55, Essen, 45147, Germany, 49 201 - 72377817.
  • Department of Central IT, Data Integration Center, University Hospital Essen, Essen, Germany.
  • Institute for Transfusion Medicine, University Hospital Essen, Essen, Germany.
  • Department of Pulmonary Medicine, University Hospital Essen - Ruhrlandklinik, Essen, Germany.
  • Department of Computer Science, University of Applied Sciences and Arts Dortmund, Dortmund, Germany.

Zitieren

APA

Idrissi-Yaghir, A., Arzideh, K., Schäfer, H., Eryilmaz, B., Bahn, M., Wen, Y., Borys, K., Hartmann, E., Schmidt, C., Pelka, O., Haubold, J., Friedrich, C. M., Nensa, F., & Hosch, R. (2025). Using a Diverse Test Suite to Assess Large Language Models on Fast Health Care Interoperability Resources Knowledge: Comparative Analysis. Journal of Medical Internet Research, 27, e73540-e73540. https://doi.org/10.2196/73540

Vancouver

Idrissi-Yaghir A, Arzideh K, Schäfer H, Eryilmaz B, Bahn M, Wen Y, et al. Using a Diverse Test Suite to Assess Large Language Models on Fast Health Care Interoperability Resources Knowledge: Comparative Analysis. Journal of Medical Internet Research. 2025;27:e73540-e73540. doi:10.2196/73540

BibTeX
@article{IdrissiYaghir2025,
  author = {Idrissi-Yaghir, Ahmad and Arzideh, Kamyar and Schäfer, Henning and Eryilmaz, Bahadir and Bahn, Mikel and Wen, Yutong and Borys, Katarzyna and Hartmann, Eva and Schmidt, Cynthia and Pelka, Obioma and Haubold, Johannes and Friedrich, Christoph M and Nensa, Felix and Hosch, René},
  title = {Using a Diverse Test Suite to Assess Large Language Models on Fast Health Care Interoperability Resources Knowledge: Comparative Analysis},
  journal = {Journal of Medical Internet Research},
  year = {2025},
  volume = {27},
  pages = {e73540--e73540},
  doi = {10.2196/73540}
}
Kostenlos starten →