EUVIMEDEUVIMED entdecken
Open Access

A scalable framework for benchmark embedding models in semantic health-care tasks

Shelly Soffer, Mahmud Omar iD, Moran Gendler iD, Benjamin S Glicksberg, Patricia Kovatch, Orly Efros iD, Robert Freeman iD, Alexander W Charney, Girish N Nadkarni, Eyal Klang

Journal of the American Medical Informatics Association · 01.12.2025 · 32 (12) · 1877-1887

DOI: 10.1093/jamia/ocaf149 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of the American Medical Informatics Association, online erschienen am 22.09.2025. Verfasst von Shelly Soffer und 9 weiteren Autor:innen. Beteiligte Einrichtung: Institute of Hematology, Davidoff Cancer Center, Rabin Medical Center, Petah Tikva, 49100, Israel. Hauptthemen laut MEDLINE: Benchmarking, Semantics, Natural Language Processing, Information Storage and Retrieval. Die Arbeit wurde bisher 6-mal zitiert (Crossref) und nennt 52 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: NCATS NIH HHS, Clinical and Translational Science.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objectives

Text embeddings are promising for semantic tasks, such as retrieval augmented generation (RAG). However, their application in health care is underexplored due to a lack of benchmarking methods. We introduce a scalable benchmarking method to test embeddings for health-care semantic tasks. Materials and Methods We evaluated 39 embedding models across 7 medical semantic similarity tasks using diverse datasets. These datasets comprised real-world patient data (from the Mount Sinai Health System and MIMIC IV), biomedical texts from PubMed, and synthetic data generated with Llama-3-70b. We first assessed semantic textual similarity (STS) by correlating the model-generated similarity scores with noise levels using Spearman rank correlation. We then reframed the same tasks as retrieval problems, evaluated by mean reciprocal rank and recall at k.

Results

In total, evaluating 2000 text pairs per 7 tasks for STS and retrieval yielded 3.28 million model assessments. Larger models (>7b parameters), such as those based on Mistral-7b and Gemma-2-9b, consistently performed well, especially in long-context tasks. The NV-Embed-v1 model (7b parameters), although top in short tasks, underperformed in long tasks. For short tasks, smaller models such as b1ade-embed (335M parameters) performed on-par to the larger models. For long retrieval tasks, the larger models significantly outperformed the smaller ones.

Discussion

The proposed benchmarking framework demonstrates scalability and flexibility, offering a structured approach to guide the selection of embedding models for a wide range of health-care tasks.

Conclusion

By matching the appropriate model with the task, the framework enables more effective deployment of embedding models, enhancing critical applications such as semantic search and retrieval-augmented generation (RAG).

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

BenchmarkingSemanticsNatural Language ProcessingInformation Storage and RetrievalHumans

Stichwörter der Autor:innen

artificial intelligencelarge language modelsmedical NLP tools

Autor:innen und Institutionen

  • Institute of Hematology, Davidoff Cancer Center, Rabin Medical Center, Petah Tikva, 49100, Israel.
  • Gray Faculty of Medical and Health Sciences, Tel Aviv University, Tel Aviv, 6997801, Israel.
  • Windreich Department of AI and Human Health, Icahn School of Medicine at Mount Sinai, New York, NY 10029, United States.
  • Department of Medicine, Icahn School of Medicine at Mount Sinai, The Hasso Plattner Institute for Digital Health at Mount Sinai, New York, NY 10019, United States.
  • Azrieli Faculty of Medicine, Bar-Ilan University, Safed, 1311502, Israel.
  • Department of Genetics and Genomic Sciences, Icahn School of Medicine at Mount Sinai, New York, NY 10029, United States.

ORCID-Profile: Mahmud Omar, Moran Gendler, Orly Efros, Robert Freeman

Förderung

  • NCATS NIH HHS
  • Clinical and Translational Science

Zitieren

APA

Soffer, S., Omar, M., Gendler, M., Glicksberg, B. S., Kovatch, P., Efros, O., Freeman, R., Charney, A. W., Nadkarni, G. N., & Klang, E. (2025). A scalable framework for benchmark embedding models in semantic health-care tasks. Journal of the American Medical Informatics Association, 32(12), 1877-1887. https://doi.org/10.1093/jamia/ocaf149

Vancouver

Soffer S, Omar M, Gendler M, Glicksberg BS, Kovatch P, Efros O, et al. A scalable framework for benchmark embedding models in semantic health-care tasks. Journal of the American Medical Informatics Association. 2025;32(12):1877-1887. doi:10.1093/jamia/ocaf149

BibTeX
@article{Soffer2025,
  author = {Soffer, Shelly and Omar, Mahmud and Gendler, Moran and Glicksberg, Benjamin S and Kovatch, Patricia and Efros, Orly and Freeman, Robert and Charney, Alexander W and Nadkarni, Girish N and Klang, Eyal},
  title = {A scalable framework for benchmark embedding models in semantic health-care tasks},
  journal = {Journal of the American Medical Informatics Association},
  year = {2025},
  volume = {32},
  number = {12},
  pages = {1877--1887},
  doi = {10.1093/jamia/ocaf149}
}
Kostenlos starten →