Evaluation of crowdsourced mortality prediction models as a framework for assessing artificial intelligence in medicine
Journal of the American Medical Informatics Association · 2023
DOI: 10.1093/jamia/ocad159 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of the American Medical Informatics Association, 2023. Verfasst von Timothy Bergquist und 21 weiteren Autor:innen. Beteiligte Einrichtung: Sage Bionetworks, Seattle, WA, United States. Hauptthemen laut MEDLINE: Crowdsourcing, Medicine. Die Arbeit wurde bisher 4-mal zitiert (Crossref) und nennt 34 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: NCATS NIH HHS, NIH HHS, Clinical and Translational Science Awards Program.
Abstract (Originalsprache)
Objective
Applications of machine learning in healthcare are of high interest and have the potential to improve patient care. Yet, the real-world accuracy of these models in clinical practice and on different patient subpopulations remains unclear. To address these important questions, we hosted a community challenge to evaluate methods that predict healthcare outcomes. We focused on the prediction of all-cause mortality as the community challenge question. Materials and methods Using a Model-to-Data framework, 345 registered participants, coalescing into 25 independent teams, spread over 3 continents and 10 countries, generated 25 accurate models all trained on a dataset of over 1.1 million patients and evaluated on patients prospectively collected over a 1-year observation of a large health system.
Results
The top performing team achieved a final area under the receiver operator curve of 0.947 (95% CI, 0.942-0.951) and an area under the precision-recall curve of 0.487 (95% CI, 0.458-0.499) on a prospectively collected patient cohort.
Discussion
Post hoc analysis after the challenge revealed that models differ in accuracy on subpopulations, delineated by race or gender, even when they are trained on the same data.
Conclusion
This is the largest community challenge focused on the evaluation of state-of-the-art machine learning methods in a healthcare system performed to date, revealing both opportunities and pitfalls of clinical AI.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Sage Bionetworks, Seattle, WA, United States.
- Department of Biomedical Informatics and Medical Education, University of Washington, Seattle, WA, United States.
- Molecular Engineering and Sciences Institute, University of Washington, Seattle, WA, United States.
- Institute of Translational Health Sciences, University of Washington, Seattle, WA, United States.
- Department of Biostatistics and Medical Informatics, University of Wisconsin-Madison, Madison, WI, United States.
- Proacta, Warsaw, Poland.
ORCID-Profile: Timothy Bergquist, Sunkyu Kim, Jaewoo Kang, Sean D Mooney, Justin Guinney
Förderung
- NCATS NIH HHS
- NIH HHS
- Clinical and Translational Science Awards Program
- ational Center for Data to Health
- National Center for Advancing Translational Sciences
- National Institutes of Health
- Institute for Translational Health Sciences
Zitieren
APA
Bergquist, T., Schaffter, T., Yan, Y., Yu, T., Prosser, J., Gao, J., Chen, G., Charzewski, Ł., Nawalany, Z., Brugere, I., Retkute, R., Prusokas, A., Prusokas, A., Choi, Y., Lee, S., Choe, J., Lee, I., Kim, S., Kang, J., …, & Patient Mortality Prediction DREAM Challenge Consortium (2023). Evaluation of crowdsourced mortality prediction models as a framework for assessing artificial intelligence in medicine. Journal of the American Medical Informatics Association. https://doi.org/10.1093/jamia/ocad159
Vancouver
Bergquist T, Schaffter T, Yan Y, Yu T, Prosser J, Gao J, et al. Evaluation of crowdsourced mortality prediction models as a framework for assessing artificial intelligence in medicine. Journal of the American Medical Informatics Association. 2023. doi:10.1093/jamia/ocad159
BibTeX
@article{Bergquist2023,
author = {Bergquist, Timothy and Schaffter, Thomas and Yan, Yao and Yu, Thomas and Prosser, Justin and Gao, Jifan and Chen, Guanhua and Charzewski, Łukasz and Nawalany, Zofia and Brugere, Ivan and Retkute, Renata and Prusokas, Alidivinas and Prusokas, Augustinas and Choi, Yonghwa and Lee, Sanghoon and Choe, Junseok and Lee, Inggeol and Kim, Sunkyu and Kang, Jaewoo and Mooney, Sean D and Guinney, Justin and Patient Mortality Prediction DREAM Challenge Consortium, Patient Mortality Prediction DREAM Challenge Consortium},
title = {Evaluation of crowdsourced mortality prediction models as a framework for assessing artificial intelligence in medicine},
journal = {Journal of the American Medical Informatics Association},
year = {2023},
doi = {10.1093/jamia/ocad159}
}
EUVIMED