A scalable framework for benchmark embedding models in semantic health-care tasks
Journal of the American Medical Informatics Association · 01.12.2025 · 32 (12) · 1877-1887
DOI: 10.1093/jamia/ocaf149 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of the American Medical Informatics Association, online erschienen am 22.09.2025. Verfasst von Shelly Soffer und 9 weiteren Autor:innen. Beteiligte Einrichtung: Institute of Hematology, Davidoff Cancer Center, Rabin Medical Center, Petah Tikva, 49100, Israel. Hauptthemen laut MEDLINE: Benchmarking, Semantics, Natural Language Processing, Information Storage and Retrieval. Die Arbeit wurde bisher 6-mal zitiert (Crossref) und nennt 52 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: NCATS NIH HHS, Clinical and Translational Science.
Abstract (Originalsprache)
Objectives
Text embeddings are promising for semantic tasks, such as retrieval augmented generation (RAG). However, their application in health care is underexplored due to a lack of benchmarking methods. We introduce a scalable benchmarking method to test embeddings for health-care semantic tasks. Materials and Methods We evaluated 39 embedding models across 7 medical semantic similarity tasks using diverse datasets. These datasets comprised real-world patient data (from the Mount Sinai Health System and MIMIC IV), biomedical texts from PubMed, and synthetic data generated with Llama-3-70b. We first assessed semantic textual similarity (STS) by correlating the model-generated similarity scores with noise levels using Spearman rank correlation. We then reframed the same tasks as retrieval problems, evaluated by mean reciprocal rank and recall at k.
Results
In total, evaluating 2000 text pairs per 7 tasks for STS and retrieval yielded 3.28 million model assessments. Larger models (>7b parameters), such as those based on Mistral-7b and Gemma-2-9b, consistently performed well, especially in long-context tasks. The NV-Embed-v1 model (7b parameters), although top in short tasks, underperformed in long tasks. For short tasks, smaller models such as b1ade-embed (335M parameters) performed on-par to the larger models. For long retrieval tasks, the larger models significantly outperformed the smaller ones.
Discussion
The proposed benchmarking framework demonstrates scalability and flexibility, offering a structured approach to guide the selection of embedding models for a wide range of health-care tasks.
Conclusion
By matching the appropriate model with the task, the framework enables more effective deployment of embedding models, enhancing critical applications such as semantic search and retrieval-augmented generation (RAG).
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Institute of Hematology, Davidoff Cancer Center, Rabin Medical Center, Petah Tikva, 49100, Israel.
- Gray Faculty of Medical and Health Sciences, Tel Aviv University, Tel Aviv, 6997801, Israel.
- Windreich Department of AI and Human Health, Icahn School of Medicine at Mount Sinai, New York, NY 10029, United States.
- Department of Medicine, Icahn School of Medicine at Mount Sinai, The Hasso Plattner Institute for Digital Health at Mount Sinai, New York, NY 10019, United States.
- Azrieli Faculty of Medicine, Bar-Ilan University, Safed, 1311502, Israel.
- Department of Genetics and Genomic Sciences, Icahn School of Medicine at Mount Sinai, New York, NY 10029, United States.
ORCID-Profile: Mahmud Omar, Moran Gendler, Orly Efros, Robert Freeman
Förderung
- NCATS NIH HHS
- Clinical and Translational Science
Zitieren
APA
Soffer, S., Omar, M., Gendler, M., Glicksberg, B. S., Kovatch, P., Efros, O., Freeman, R., Charney, A. W., Nadkarni, G. N., & Klang, E. (2025). A scalable framework for benchmark embedding models in semantic health-care tasks. Journal of the American Medical Informatics Association, 32(12), 1877-1887. https://doi.org/10.1093/jamia/ocaf149
Vancouver
Soffer S, Omar M, Gendler M, Glicksberg BS, Kovatch P, Efros O, et al. A scalable framework for benchmark embedding models in semantic health-care tasks. Journal of the American Medical Informatics Association. 2025;32(12):1877-1887. doi:10.1093/jamia/ocaf149
BibTeX
@article{Soffer2025,
author = {Soffer, Shelly and Omar, Mahmud and Gendler, Moran and Glicksberg, Benjamin S and Kovatch, Patricia and Efros, Orly and Freeman, Robert and Charney, Alexander W and Nadkarni, Girish N and Klang, Eyal},
title = {A scalable framework for benchmark embedding models in semantic health-care tasks},
journal = {Journal of the American Medical Informatics Association},
year = {2025},
volume = {32},
number = {12},
pages = {1877--1887},
doi = {10.1093/jamia/ocaf149}
}
EUVIMED