Generating Real‐World Evidence From AI Deployed in Health Systems
Learning Health Systems · 2027-01 · 11 (1)
DOI: 10.1002/lrh2.70131 ↗Kurz gefasst
Fachartikel in der Zeitschrift Learning Health Systems, online erschienen am 04.11.2026. Verfasst von Lily Poursoltan und 8 weiteren Autor:innen. Beteiligte Einrichtung: Rady School of Management University of California San Diego California USA. Die Arbeit nennt 24 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Background
Health systems increasingly deploy predictive and generative artificial intelligence ( AI ) to improve care delivery and outcomes. Predictive AI models estimate the risk of future events to inform targeted interventions, while generative AI systems extract, summarize, and communicate information and may automate selected tasks. Routine deployment generates clinical, operational, and interaction data that can support evaluation of the effectiveness and safety of AI ‐enabled workflows.
Methods
We review approaches to generating real‐world evidence ( RWE ) from routine deployment of predictive and generative AI across four implementation configurations: silent deployment, human‐in‐the‐loop ( HITL ), human‐on‐the‐loop ( HOTL ), and automated. For each configuration, we describe the evaluation questions, routinely generated data, and feasible study designs.
Results
Each implementation configuration supports different forms of evidence about model performance, workflow effectiveness, human oversight, and safety. For both predictive and generative AI , silent deployment supports assessment of local performance and subgroup differences. Once predictive AI drives action, evaluating the effectiveness of an implemented workflow ( HITL , HOTL , or automated) generally requires causal inference approaches. For generative AI , HITL implementations can generate data on edits, overrides, time on task, usability, cognitive burden, and safety‐relevant failure modes. HOTL and automated generative AI implementations can generate data on escalation performance, human follow‐up work, downstream outcomes, and safety signals.
Conclusion
AI implementation should be paired with methods for generating RWE on the effectiveness and safety of AI ‐enabled workflows. In HOTL and automated configurations, escalation performance and residual human workload become particularly important components of evaluation.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Rady School of Management University of California San Diego California USA.
- Joan and Irwin Jacobs Center for Health Innovation UC San Diego Health San Diego California USA.
- Division of Biomedical Informatics, Department of Medicine University of California San Diego California USA.
- Division of General Internal Medicine, Department of Medicine University of California San Diego California USA.
- Department of Electrical and Computer Engineering University of California San Diego California USA.
- Department of Biology University of California San Diego California USA.
ORCID-Profile: Lily Poursoltan, Eileen Kim, Poorva Satish Bedmutha, Mattheus Ramsis, Karandeep Singh
Zitieren
APA
Poursoltan, L., Boussina, A., Cao, J., Kim, E., Bedmutha, P. S., Majmudar, H., Ramsis, M., Khalessi, A., & Singh, K. (2027). Generating Real‐World Evidence From AI Deployed in Health Systems. Learning Health Systems, 11(1). https://doi.org/10.1002/lrh2.70131
Vancouver
Poursoltan L, Boussina A, Cao J, Kim E, Bedmutha PS, Majmudar H, et al. Generating Real‐World Evidence From AI Deployed in Health Systems. Learning Health Systems. 2027;11(1). doi:10.1002/lrh2.70131
BibTeX
@article{Poursoltan2027,
author = {Poursoltan, Lily and Boussina, Aaron and Cao, Jie and Kim, Eileen and Bedmutha, Poorva Satish and Majmudar, Hetvi and Ramsis, Mattheus and Khalessi, Alexander and Singh, Karandeep},
title = {Generating Real‐World Evidence From AI Deployed in Health Systems},
journal = {Learning Health Systems},
year = {2027},
volume = {11},
number = {1},
doi = {10.1002/lrh2.70131}
}
EUVIMED