EUVIMEDEUVIMED entdecken
Open Access

Evaluation of crowdsourced mortality prediction models as a framework for assessing artificial intelligence in medicine

Timothy Bergquist iD, Thomas Schaffter, Yao Yan, Thomas Yu, Justin Prosser, Jifan Gao, Guanhua Chen, Łukasz Charzewski, Zofia Nawalany, Ivan Brugere, Renata Retkute, Alidivinas Prusokas und 10 weitere

Journal of the American Medical Informatics Association · 2023

DOI: 10.1093/jamia/ocad159 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of the American Medical Informatics Association, 2023. Verfasst von Timothy Bergquist und 21 weiteren Autor:innen. Beteiligte Einrichtung: Sage Bionetworks, Seattle, WA, United States. Hauptthemen laut MEDLINE: Crowdsourcing, Medicine. Die Arbeit wurde bisher 4-mal zitiert (Crossref) und nennt 34 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: NCATS NIH HHS, NIH HHS, Clinical and Translational Science Awards Program.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objective

Applications of machine learning in healthcare are of high interest and have the potential to improve patient care. Yet, the real-world accuracy of these models in clinical practice and on different patient subpopulations remains unclear. To address these important questions, we hosted a community challenge to evaluate methods that predict healthcare outcomes. We focused on the prediction of all-cause mortality as the community challenge question. Materials and methods Using a Model-to-Data framework, 345 registered participants, coalescing into 25 independent teams, spread over 3 continents and 10 countries, generated 25 accurate models all trained on a dataset of over 1.1 million patients and evaluated on patients prospectively collected over a 1-year observation of a large health system.

Results

The top performing team achieved a final area under the receiver operator curve of 0.947 (95% CI, 0.942-0.951) and an area under the precision-recall curve of 0.487 (95% CI, 0.458-0.499) on a prospectively collected patient cohort.

Discussion

Post hoc analysis after the challenge revealed that models differ in accuracy on subpopulations, delineated by race or gender, even when they are trained on the same data.

Conclusion

This is the largest community challenge focused on the evaluation of state-of-the-art machine learning methods in a healthcare system performed to date, revealing both opportunities and pitfalls of clinical AI.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

CrowdsourcingMedicineHumansArtificial IntelligenceMachine LearningAlgorithms

Stichwörter der Autor:innen

evaluationhealth informaticsmachine learning

Autor:innen und Institutionen

  • Sage Bionetworks, Seattle, WA, United States.
  • Department of Biomedical Informatics and Medical Education, University of Washington, Seattle, WA, United States.
  • Molecular Engineering and Sciences Institute, University of Washington, Seattle, WA, United States.
  • Institute of Translational Health Sciences, University of Washington, Seattle, WA, United States.
  • Department of Biostatistics and Medical Informatics, University of Wisconsin-Madison, Madison, WI, United States.
  • Proacta, Warsaw, Poland.

ORCID-Profile: Timothy Bergquist, Sunkyu Kim, Jaewoo Kang, Sean D Mooney, Justin Guinney

Förderung

  • NCATS NIH HHS
  • NIH HHS
  • Clinical and Translational Science Awards Program
  • ational Center for Data to Health
  • National Center for Advancing Translational Sciences
  • National Institutes of Health
  • Institute for Translational Health Sciences

Zitieren

APA

Bergquist, T., Schaffter, T., Yan, Y., Yu, T., Prosser, J., Gao, J., Chen, G., Charzewski, Ł., Nawalany, Z., Brugere, I., Retkute, R., Prusokas, A., Prusokas, A., Choi, Y., Lee, S., Choe, J., Lee, I., Kim, S., Kang, J., …, & Patient Mortality Prediction DREAM Challenge Consortium (2023). Evaluation of crowdsourced mortality prediction models as a framework for assessing artificial intelligence in medicine. Journal of the American Medical Informatics Association. https://doi.org/10.1093/jamia/ocad159

Vancouver

Bergquist T, Schaffter T, Yan Y, Yu T, Prosser J, Gao J, et al. Evaluation of crowdsourced mortality prediction models as a framework for assessing artificial intelligence in medicine. Journal of the American Medical Informatics Association. 2023. doi:10.1093/jamia/ocad159

BibTeX
@article{Bergquist2023,
  author = {Bergquist, Timothy and Schaffter, Thomas and Yan, Yao and Yu, Thomas and Prosser, Justin and Gao, Jifan and Chen, Guanhua and Charzewski, Łukasz and Nawalany, Zofia and Brugere, Ivan and Retkute, Renata and Prusokas, Alidivinas and Prusokas, Augustinas and Choi, Yonghwa and Lee, Sanghoon and Choe, Junseok and Lee, Inggeol and Kim, Sunkyu and Kang, Jaewoo and Mooney, Sean D and Guinney, Justin and Patient Mortality Prediction DREAM Challenge Consortium, Patient Mortality Prediction DREAM Challenge Consortium},
  title = {Evaluation of crowdsourced mortality prediction models as a framework for assessing artificial intelligence in medicine},
  journal = {Journal of the American Medical Informatics Association},
  year = {2023},
  doi = {10.1093/jamia/ocad159}
}
Kostenlos starten →