EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Moving beyond the benchmarks: Five foundational principles for meaningful AI evaluation in healthcare

Catherine G Bielick iD, Aya Awwad, Jacob Ellen, Laleh Jalilian, Liam G McCoy, Vishala Mishra, Esli Osmanlliu, Stephen R Pfohl, Leo A Celi

PLOS Digital Health · 26.05.2026 · 5 (5) · e0001115

DOI: 10.1371/journal.pdig.0001115 ↗

Kurz gefasst

Fachartikel in der Zeitschrift PLOS Digital Health, online erschienen am 26.05.2026. Verfasst von Catherine G Bielick und 8 weiteren Autor:innen. Beteiligte Einrichtung: Division of Infectious Diseases, Beth Israel Deaconess Medical Center, Boston, Massachusetts, United States of America. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 35 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: NIAID NIH HHS, National Institutes of Health, Foundation for the National Institutes of Health.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Rapid integration of Large Language Models (LLMs) into healthcare has exposed a critical disconnect between technical performance and clinical value. While state-of-the-art models achieve impressive scores on standardized medical examinations, their real-world impact remains limited, with few models progressing to successful clinical integration. This disconnect persists, in part, due to a proliferation of evaluation practices that prioritize static, decontextualized benchmarks. To help address this gap, we propose five foundational principles to guide contextually appropriate evaluations of healthcare AI: Local (grounded in specific deployment contexts), Task-specific (aligned with intended clinical use), Agile (continuously adaptive), Reflective (acknowledging limitations and inherent value-sensitivity), and Community-partnered (centering affected voices). We argue that emphasis on these principles can help shift evaluation practice towards assessment of artificial intelligence. This reorientation is essential for developing healthcare AI that not only performs well technically, but also can meaningfully improve patient care, serve communities for defined purposes, and mitigate (rather than exacerbate) health disparities.

Autor:innen und Institutionen

  • Division of Infectious Diseases, Beth Israel Deaconess Medical Center, Boston, Massachusetts, United States of America.
  • Harvard Medical School, Boston, Massachusetts, United States of America‌‌.
  • Metabolism Unit, Massachusetts General Hospital, Boston, Massachusetts, United States of America.
  • David Geffen School of Medicine, University of California, Los Angeles, Los Angeles, California, United States of America.
  • Division of Neurology, Faculty of Medicine and Dentistry, University of Alberta, Edmonton, Alberta, Canada.
  • Department of Medicine, Beth Israel Deaconess Medical Center, Boston, Massachusetts, United States of America.

ORCID-Profile: Catherine G Bielick

Förderung

  • NIAID NIH HHS
  • National Institutes of Health
  • Foundation for the National Institutes of Health
  • National Science Foundation
  • Food Industry Research and Development Institute
  • Fonds de Recherche du Québec - Santé
  • 2024 Detweiler Travelling Fellowship
  • Division of Intramural Research, National Institute of Allergy and Infectious Diseases

Zitieren

APA

Bielick, C. G., Awwad, A., Ellen, J., Jalilian, L., McCoy, L. G., Mishra, V., Osmanlliu, E., Pfohl, S. R., & Celi, L. A. (2026). Moving beyond the benchmarks: Five foundational principles for meaningful AI evaluation in healthcare. PLOS Digital Health, 5(5), e0001115. https://doi.org/10.1371/journal.pdig.0001115

Vancouver

Bielick CG, Awwad A, Ellen J, Jalilian L, McCoy LG, Mishra V, et al. Moving beyond the benchmarks: Five foundational principles for meaningful AI evaluation in healthcare. PLOS Digital Health. 2026;5(5):e0001115. doi:10.1371/journal.pdig.0001115

BibTeX
@article{Bielick2026,
  author = {Bielick, Catherine G and Awwad, Aya and Ellen, Jacob and Jalilian, Laleh and McCoy, Liam G and Mishra, Vishala and Osmanlliu, Esli and Pfohl, Stephen R and Celi, Leo A},
  title = {Moving beyond the benchmarks: Five foundational principles for meaningful AI evaluation in healthcare},
  journal = {PLOS Digital Health},
  year = {2026},
  volume = {5},
  number = {5},
  pages = {e0001115},
  doi = {10.1371/journal.pdig.0001115}
}
Kostenlos starten →