EUVIMEDEUVIMED entdecken
VergleichsstudieEvaluationsstudieOpen Access

Evaluating retrieval-augmented generation versus long-context input for clinical reasoning over electronic health records

Skatje Myers iD, Dmitriy Dligach iD, Timothy A Miller iD, Samantha Barr, James Landefeld, Yanjun Gao iD, Matthew M Churpek, Anoop Mayampurath, Majid Afshar iD

Journal of the American Medical Informatics Association · 2026

DOI: 10.1093/jamia/ocag139 ↗

Kurz gefasst

Vergleichsstudie in der Zeitschrift Journal of the American Medical Informatics Association, 2026. Verfasst von Skatje Myers und 8 weiteren Autor:innen. Beteiligte Einrichtung: Department of Medicine, University of Wisconsin-Madison, Madison, WI 53726, United States. Hauptthemen laut MEDLINE: Electronic Health Records, Information Storage and Retrieval, Natural Language Processing. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 26 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: NIGMS NIH HHS, NIH HHS, National Library of Medicine of the National Institutes of Health.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objective

To evaluate whether retrieval-augmented generation (RAG) can serve as an efficient alternative to long-context prompting for clinical reasoning over electronic health records (EHRs). Materials and Methods We defined 3 EHR-based tasks that are replicable across health systems and vary in reasoning complexity: (1) extracting imaging procedures (modality, date, and anatomic site), (2) generating timelines of therapeutic antibiotic use, and (3) identifying the key diagnoses for a hospitalization. Using real inpatient clinical notes from a US academic health system, we evaluated 3 large language models (GPT-5.4-mini, Mistral Medium 3, DeepSeek V3.1) with varying amounts of provided context, comparing targeted retrieval to using the most recent clinical notes.

Results

For Imaging Procedures, RAG strongly outperformed recent-note inputs and exceeded long-context performance (by 0.17-9.83 F1 across all models) using fewer than 8K tokens. Similar benefits were observed for Antibiotic Timelines, where <8K of retrieved tokens matched long-context recent-notes performance (between -3.26 and +3.24 Jaccard). Error analysis revealed that missing information in the clinical notes—often due to inter-hospital transfers—limited performance to some extent. However, performance on the Diagnosis Generation task remains largely static across methods and models.

Discussion

RAG demonstrated strong token efficiency across tasks, with the clearest and most consistent gains observed for imaging extraction and antibiotic timeline reconstruction. Diagnosis generation proved the most challenging task, suggesting ceiling effects imposed by documentation variability and evaluation constraints.

Conclusion

Our results suggest that RAG remains a competitive and efficient approach for clinical tasks over large amounts of EHR, even as newer models become capable of handling increasingly longer amounts of text.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Electronic Health RecordsInformation Storage and RetrievalNatural Language ProcessingLarge Language ModelsHumansGenerative Artificial IntelligenceDiagnostic ImagingAnti-Bacterial Agents

Stichwörter der Autor:innen

electronic health recordsnatural language processingretrieval-augmented generation

Substanzen

Anti-Bacterial Agents

Autor:innen und Institutionen

  • Department of Medicine, University of Wisconsin-Madison, Madison, WI 53726, United States.
  • Department of Computer Science, Loyola University Chicago, Chicago, IL 60626, United States.
  • Computational Health Informatics Program, Boston Children's Hospital, Boston, MA 02215, United States.
  • Department of Pediatrics, Harvard Medical School, Boston, MA 02115, United States.
  • Department of Biomedical Informatics, University of Colorado-Anschutz, Aurora, CO 80045, United States.
  • Department of Biostatistics and Medical Informatics, University of Wisconsin-Madison, Madison, WI 53726, United States.

ORCID-Profile: Skatje Myers, Dmitriy Dligach, Timothy A Miller, Yanjun Gao, Majid Afshar

Förderung

  • NIGMS NIH HHS
  • NIH HHS
  • National Library of Medicine of the National Institutes of Health
  • NLM NIH HHS
  • National Library of Medicine
  • National Institutes of Health
  • National Institute of General Medical Sciences

Zitieren

APA

Myers, S., Dligach, D., Miller, T. A., Barr, S., Landefeld, J., Gao, Y., Churpek, M. M., Mayampurath, A., & Afshar, M. (2026). Evaluating retrieval-augmented generation versus long-context input for clinical reasoning over electronic health records. Journal of the American Medical Informatics Association. https://doi.org/10.1093/jamia/ocag139

Vancouver

Myers S, Dligach D, Miller TA, Barr S, Landefeld J, Gao Y, et al. Evaluating retrieval-augmented generation versus long-context input for clinical reasoning over electronic health records. Journal of the American Medical Informatics Association. 2026. doi:10.1093/jamia/ocag139

BibTeX
@article{Myers2026,
  author = {Myers, Skatje and Dligach, Dmitriy and Miller, Timothy A and Barr, Samantha and Landefeld, James and Gao, Yanjun and Churpek, Matthew M and Mayampurath, Anoop and Afshar, Majid},
  title = {Evaluating retrieval-augmented generation versus long-context input for clinical reasoning over electronic health records},
  journal = {Journal of the American Medical Informatics Association},
  year = {2026},
  doi = {10.1093/jamia/ocag139}
}
Kostenlos starten →