Evaluating retrieval-augmented generation versus long-context input for clinical reasoning over electronic health records
Journal of the American Medical Informatics Association · 2026
DOI: 10.1093/jamia/ocag139 ↗Kurz gefasst
Vergleichsstudie in der Zeitschrift Journal of the American Medical Informatics Association, 2026. Verfasst von Skatje Myers und 8 weiteren Autor:innen. Beteiligte Einrichtung: Department of Medicine, University of Wisconsin-Madison, Madison, WI 53726, United States. Hauptthemen laut MEDLINE: Electronic Health Records, Information Storage and Retrieval, Natural Language Processing. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 26 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: NIGMS NIH HHS, NIH HHS, National Library of Medicine of the National Institutes of Health.
Abstract (Originalsprache)
Objective
To evaluate whether retrieval-augmented generation (RAG) can serve as an efficient alternative to long-context prompting for clinical reasoning over electronic health records (EHRs). Materials and Methods We defined 3 EHR-based tasks that are replicable across health systems and vary in reasoning complexity: (1) extracting imaging procedures (modality, date, and anatomic site), (2) generating timelines of therapeutic antibiotic use, and (3) identifying the key diagnoses for a hospitalization. Using real inpatient clinical notes from a US academic health system, we evaluated 3 large language models (GPT-5.4-mini, Mistral Medium 3, DeepSeek V3.1) with varying amounts of provided context, comparing targeted retrieval to using the most recent clinical notes.
Results
For Imaging Procedures, RAG strongly outperformed recent-note inputs and exceeded long-context performance (by 0.17-9.83 F1 across all models) using fewer than 8K tokens. Similar benefits were observed for Antibiotic Timelines, where <8K of retrieved tokens matched long-context recent-notes performance (between -3.26 and +3.24 Jaccard). Error analysis revealed that missing information in the clinical notes—often due to inter-hospital transfers—limited performance to some extent. However, performance on the Diagnosis Generation task remains largely static across methods and models.
Discussion
RAG demonstrated strong token efficiency across tasks, with the clearest and most consistent gains observed for imaging extraction and antibiotic timeline reconstruction. Diagnosis generation proved the most challenging task, suggesting ceiling effects imposed by documentation variability and evaluation constraints.
Conclusion
Our results suggest that RAG remains a competitive and efficient approach for clinical tasks over large amounts of EHR, even as newer models become capable of handling increasingly longer amounts of text.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Substanzen
Autor:innen und Institutionen
- Department of Medicine, University of Wisconsin-Madison, Madison, WI 53726, United States.
- Department of Computer Science, Loyola University Chicago, Chicago, IL 60626, United States.
- Computational Health Informatics Program, Boston Children's Hospital, Boston, MA 02215, United States.
- Department of Pediatrics, Harvard Medical School, Boston, MA 02115, United States.
- Department of Biomedical Informatics, University of Colorado-Anschutz, Aurora, CO 80045, United States.
- Department of Biostatistics and Medical Informatics, University of Wisconsin-Madison, Madison, WI 53726, United States.
ORCID-Profile: Skatje Myers, Dmitriy Dligach, Timothy A Miller, Yanjun Gao, Majid Afshar
Förderung
- NIGMS NIH HHS
- NIH HHS
- National Library of Medicine of the National Institutes of Health
- NLM NIH HHS
- National Library of Medicine
- National Institutes of Health
- National Institute of General Medical Sciences
Zitieren
APA
Myers, S., Dligach, D., Miller, T. A., Barr, S., Landefeld, J., Gao, Y., Churpek, M. M., Mayampurath, A., & Afshar, M. (2026). Evaluating retrieval-augmented generation versus long-context input for clinical reasoning over electronic health records. Journal of the American Medical Informatics Association. https://doi.org/10.1093/jamia/ocag139
Vancouver
Myers S, Dligach D, Miller TA, Barr S, Landefeld J, Gao Y, et al. Evaluating retrieval-augmented generation versus long-context input for clinical reasoning over electronic health records. Journal of the American Medical Informatics Association. 2026. doi:10.1093/jamia/ocag139
BibTeX
@article{Myers2026,
author = {Myers, Skatje and Dligach, Dmitriy and Miller, Timothy A and Barr, Samantha and Landefeld, James and Gao, Yanjun and Churpek, Matthew M and Mayampurath, Anoop and Afshar, Majid},
title = {Evaluating retrieval-augmented generation versus long-context input for clinical reasoning over electronic health records},
journal = {Journal of the American Medical Informatics Association},
year = {2026},
doi = {10.1093/jamia/ocag139}
}
EUVIMED