Cross-Site Stability of Embedding Model Rankings for Known-Item Retrieval From Clinical Notes: Two-Corpus Comparative Evaluation Study
JMIR AI · 11.11.2026 · 5 · e109305
DOI: 10.2196/109305 ↗Kurz gefasst
Fachartikel in der Zeitschrift JMIR AI, online erschienen am 11.11.2026. Verfasst von Yngve Mikkelsen. Beteiligte Einrichtung: Saïd Business School, University of Oxford, Oxford, England, United Kingdom. Die Arbeit nennt 29 Literaturangaben.
Abstract (Originalsprache)
Background
Retrieval-augmented generation over clinical text depends on the embedding model chosen—a choice that interacts with documentation genre. Cross-site stability of embedding-model rankings has not been evaluated on real electronic health record (EHR) narratives with genre held constant; public benchmarks are predominantly single-source, and multi-institution evaluations compare models and query formulations within tasks rather than ranking stability across institutions.
Objective
The aim of the study is to test whether embedding model rankings transfer across 2 institutional EHR corpora, with genre held constant; to examine the contributions of model, site/corpus, and genre and their dependence on scoring; and to test whether the dominant contributor depends on model-panel composition.
Methods
We compared 13 embedding models on a known-item passage-to-source-document task, each query a deterministic 2-sentence excerpt from its target, in a 2×2 design crossing site/corpus (Beth Israel Deaconess Medical Center via MIMIC-IV-Note [Medical Information Mart for Intensive Care IV Clinical Notes Module] and University of California, San Francisco via ER-Reason) with genre (discharge summary and imaging report). Query sentences were removed from their targets before indexing to prevent verbatim overlap, notes were deduplicated after normalization, models used their recommended pooling, long documents were scored as overlapping chunks by maximum similarity, and cell sizes were matched (N=1235 per cell). Effectiveness was measured by mean reciprocal rank at 10, rank transfer by Kendall τ and selection regret, and variance by factorial decomposition (η2). Transfer, regret, and the contrastive-panel decomposition carry 95% patient-clustered bootstrap intervals.
Results
Among the 8 contrastive models, cross-site Kendall τ was 0.86 (95% CI 0.64-1.00) for discharge and 0.64 (95% CI 0.36-0.86) for imaging. The model×site interaction was small under every scoring choice (η2=0.005-0.029); the site/corpus main effect was negligible under best-chunk scoring (η2=0.000) but scoring-dependent (0.172 under mean-pooling). Cross-site selection regret was 0 for imaging and up to 0.033 for discharge (~10% of the destination’s best mean reciprocal rank; intervals 0-0.071). Adding 5 nonretrieval masked-language-model comparators moved the largest term to model (η2=0.874): the dominant-factor question is panel-sensitive, while the qualitative transfer conclusion was unchanged.
Conclusions
For known-item retrieval from clinical notes at 2 US academic medical centers, embedding-model rankings showed positive cross-site concordance, moderate for imaging and strong for discharge summaries—with intervals wide enough that direction, not magnitude, is the finding. Which factor dominates absolute effectiveness depends on the panel evaluated: primary contrastive-panel scoring favors genre, whereas adding nonretrieval comparators makes model choice dominant; so, claims about major determinants are panel-specific. Cross-site model selection incurred zero to modest performance loss. When documentation genre and retrieval configuration are matched, model-selection results from one institutional EHR corpus may be a useful starting point for another, although local confirmation remains warranted.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Saïd Business School, University of Oxford, Oxford, England, United Kingdom.
ORCID-Profile: Yngve Mikkelsen
Zitieren
APA
Mikkelsen, Y. (2026). Cross-Site Stability of Embedding Model Rankings for Known-Item Retrieval From Clinical Notes: Two-Corpus Comparative Evaluation Study. JMIR AI, 5, e109305. https://doi.org/10.2196/109305
Vancouver
Mikkelsen Y. Cross-Site Stability of Embedding Model Rankings for Known-Item Retrieval From Clinical Notes: Two-Corpus Comparative Evaluation Study. JMIR AI. 2026;5:e109305. doi:10.2196/109305
BibTeX
@article{Mikkelsen2026,
author = {Mikkelsen, Yngve},
title = {Cross-Site Stability of Embedding Model Rankings for Known-Item Retrieval From Clinical Notes: Two-Corpus Comparative Evaluation Study},
journal = {JMIR AI},
year = {2026},
volume = {5},
pages = {e109305},
doi = {10.2196/109305}
}
EUVIMED