EUVIMEDEUVIMED entdecken
EvaluationsstudieOpen Access

Improving Retrieval Augmented Generation for Health Care by Fine-Tuning Clinical Embedding Models: Development and Evaluation Study

Kamyar Arzideh, Henning Schäfer, Ahmad Idrissi-Yaghir, Cynthia Sabrina Schmidt, Bahadir Eryilmaz, Mikel Bahn, Amin T Turki, Olivia Barbara Pollok, Eva Maria Hartmann, Philipp Winnekens, Katarzyna Borys, Johannes Haubold und 2 weitere

Journal of Medical Internet Research · 2026

DOI: 10.2196/82997 ↗

Kurz gefasst

Evaluationsstudie in der Zeitschrift Journal of Medical Internet Research, 2026. Verfasst von Kamyar Arzideh und 13 weiteren Autor:innen. Beteiligte Einrichtung: Institute for Artificial Intelligence in Medicine,, University Hospital Essen, Essen, Germany. Hauptthemen laut MEDLINE: Information Storage and Retrieval, Electronic Health Records, Delivery of Health Care. Die Arbeit wurde bisher 7-mal zitiert (Crossref) und nennt 42 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Embedding models are critical components of Retrieval Augmented Generation (RAG) systems for retrieving and searching unstructured medical data. However, existing models are predominantly trained on publicly available English datasets, limiting their effectiveness in non-English health care settings. More importantly, these models lack training on real-world clinical documents, leading to inaccurate context retrieval when integrated into RAG systems for health care applications. This gap is particularly pronounced in specialized medical documentation containing domain-specific terminology, abbreviations, and nuanced clinical language.

Objective

This retrospective study aimed to develop and validate embedding models specifically trained on real-world clinical documents from multiple medical specialties to improve medical information retrieval (IR) and RAG system performance in both German and English language contexts.

Methods

We fine-tuned embedding models, so-called sentence transformers, using the multilingual-e5-large architecture as a foundation. Training data consisted of approximately 11 million question-answer pairs synthetically generated from 400,000 diverse clinical documents from a large German tertiary hospital, spanning 163,840 patients and 282,728 clinical cases between 2018 and 2023. The SauerkrautLM-SOLAR-Instruct large language model generated medically relevant questions and corresponding answers for each document. The dataset was additionally pseudonymized and translated into English to aim for broader applicability. Models were evaluated in 2 distinct scenarios: IR using questions with multiple relevant passages, and RAG system performance in both cross-patient and patient-centered contexts.

Results

In the IR evaluation, the fine-tuned miracle model achieved a mAP@100 of 0.27, outperforming the multilingual-e5-large baseline (0.14) and state-of-the-art models such as bge-m3 (0.11). In the RAG evaluation, the model demonstrated robust performance comparable with the baseline in the constrained patient-centered scenario (BERTScore F1 0.781 vs 0.778) and showed moderate improvements in the unconstrained cross-patient setting (BLEURT 0.56 vs 0.53). Notably, the model trained on pseudonymized data achieved comparable retrieval performance (mAP@100 0.25) and the highest scores for patient-centered contextual precision (0.93). Performance gains were robust in the German dataset, while the translated English model demonstrated promising results as a proof of concept for cross-lingual transfer.

Conclusions

By leveraging a comprehensive real-world dataset spanning multiple medical specialties and using large language models for synthetic question generation, we successfully created and validated domain-specific embedding models. These models can improve medical IR in large-scale search spaces and perform competitively in constrained RAG applications. By publishing the models trained on pseudonymized data, other health care institutions can integrate or adapt these embedding models to their needs. This work establishes a reproducible framework for developing domain-specific clinical embedding models, with the potential to improve data retrieval in medical settings.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Information Storage and RetrievalElectronic Health RecordsDelivery of Health CareHumansRetrospective StudiesLarge Language ModelsNatural Language Processing

Stichwörter der Autor:innen

LLMNLPRAGRetrieval Augmented Generationinformation retrievallarge language modelsnatural language processing

Autor:innen und Institutionen

  • Institute for Artificial Intelligence in Medicine,, University Hospital Essen, Essen, Germany.
  • Central IT Department, Data Integration Center, University Hospital Essen, Girardetstr. 2, Essen, 45131, Germany, 49 0231-77816.
  • Institute for Transfusion Medicine, University Hospital Essen, Essen, Germany.
  • Institute of Diagnostic and Interventional Radiology and Neuroradiology, University Hospital Essen, Essen, Germany.
  • Center of Sleep and Telemedicine, University Hospital Essen, Ruhrlandklinik, Essen, Germany.
  • Department of Hematology and Oncology, Ruhr-University Bochum, Marienhospital University Hospital, Bochum, Germany.

Zitieren

APA

Arzideh, K., Schäfer, H., Idrissi-Yaghir, A., Schmidt, C. S., Eryilmaz, B., Bahn, M., Turki, A. T., Pollok, O. B., Hartmann, E. M., Winnekens, P., Borys, K., Haubold, J., Nensa, F., & Hosch, R. (2026). Improving Retrieval Augmented Generation for Health Care by Fine-Tuning Clinical Embedding Models: Development and Evaluation Study. Journal of Medical Internet Research. https://doi.org/10.2196/82997

Vancouver

Arzideh K, Schäfer H, Idrissi-Yaghir A, Schmidt CS, Eryilmaz B, Bahn M, et al. Improving Retrieval Augmented Generation for Health Care by Fine-Tuning Clinical Embedding Models: Development and Evaluation Study. Journal of Medical Internet Research. 2026. doi:10.2196/82997

BibTeX
@article{Arzideh2026,
  author = {Arzideh, Kamyar and Schäfer, Henning and Idrissi-Yaghir, Ahmad and Schmidt, Cynthia Sabrina and Eryilmaz, Bahadir and Bahn, Mikel and Turki, Amin T and Pollok, Olivia Barbara and Hartmann, Eva Maria and Winnekens, Philipp and Borys, Katarzyna and Haubold, Johannes and Nensa, Felix and Hosch, René},
  title = {Improving Retrieval Augmented Generation for Health Care by Fine-Tuning Clinical Embedding Models: Development and Evaluation Study},
  journal = {Journal of Medical Internet Research},
  year = {2026},
  doi = {10.2196/82997}
}
Kostenlos starten →