Clinical Context Variables Collectively Rival Model Choice in Embedding-Based Retrieval: Multi-Corpus Benchmark Study
JMIR Medical Informatics · 2026-05-07 · 14 · e94241
DOI: 10.2196/94241 ↗At a glance
Journal article in JMIR Medical Informatics, published online on 2026-05-07. Written by Yngve Mikkelsen. Institution: Saïd Business School, University of Oxford, Oxford, England, United Kingdom. Major MEDLINE topics: Benchmarking, Information Storage and Retrieval, Natural Language Processing. Cited 8 times so far (Crossref); 28 references. The full text is freely available in PubMed Central.
Abstract (original language)
Background
Retrieval-augmented generation (RAG) systems increasingly support clinical decision-making by grounding large language model outputs in verifiable evidence. The retrieval component is foundational: if the correct document is not retrieved, downstream generation cannot recover it. Despite this, embedding model selection for clinical RAG remains guided by general-domain benchmarks with limited clinical coverage. Given the heterogeneity of clinical documentation across institutions, specialties, and electronic health record systems, it is unclear whether general-domain model rankings generalize to clinical retrieval tasks.
Objective
This study evaluated whether clinical context variables, corpus type (encompassing differences in document length, medical specialty, and structural characteristics), and query format have effects on retrieval performance comparable to or exceeding those of embedding model choice.
Methods
Ten primary embedding models plus two ablation variants and a BM25 lexical baseline (13 retrieval configurations total) were benchmarked on three clinical corpora (MTSamples medical transcriptions, n=500; PMC-Patients case reports, n=500; Mistral-7B-generated synthetic clinical notes, n=500). Twelve embedding configurations were evaluated across 3 corpora × 2 query formats (keyword vs natural language) × 4 chunking strategies, yielding 294 experimental conditions. Primary metrics included MRR@10, P@1, Recall@10/20/50/100, and NDCG@10, with bootstrap confidence intervals. Relative factor contributions were quantified using factorial ANOVA with η² effect sizes, including all two-way interactions.
Results
In a factorial ANOVA across 288 balanced embedding conditions, embedding model choice explained 40.8% of variance in MRR@10 (η²=0.408), corpus type 24.6%, and query format 19.2%. Chunking strategy explained minimal variance (η²=0.002). The model × query format interaction (η²=0.029, P
Subjects
MeSH terms (MEDLINE)
Author keywords
Authors and institutions
- Saïd Business School, University of Oxford, Oxford, England, United Kingdom.
ORCID profiles: Yngve Mikkelsen
Cite
APA
Mikkelsen, Y. (2026). Clinical Context Variables Collectively Rival Model Choice in Embedding-Based Retrieval: Multi-Corpus Benchmark Study. JMIR Medical Informatics, 14, e94241. https://doi.org/10.2196/94241
Vancouver
Mikkelsen Y. Clinical Context Variables Collectively Rival Model Choice in Embedding-Based Retrieval: Multi-Corpus Benchmark Study. JMIR Medical Informatics. 2026;14:e94241. doi:10.2196/94241
BibTeX
@article{Mikkelsen2026,
author = {Mikkelsen, Yngve},
title = {Clinical Context Variables Collectively Rival Model Choice in Embedding-Based Retrieval: Multi-Corpus Benchmark Study},
journal = {JMIR Medical Informatics},
year = {2026},
volume = {14},
pages = {e94241},
doi = {10.2196/94241}
}
EUVIMED