Multisite evaluation of prediction models for emergency department crowding before and during the COVID-19 pandemic
Journal of the American Medical Informatics Association · 2023
DOI: 10.1093/jamia/ocac214 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of the American Medical Informatics Association, 2023. Verfasst von Ari J Smith und 8 weiteren Autor:innen. Beteiligte Einrichtung: Department of Industrial and Systems Engineering, University of Wisconsin-Madison, Madison, Wisconsin, USA. Hauptthemen laut MEDLINE: COVID-19, Emergency Service, Hospital, Crowding. Die Arbeit wurde bisher 7-mal zitiert (Crossref) und nennt 32 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: UW School of Medicine and Public Health, the Wisconsin Partnership Program.
Abstract (Originalsprache)
Objective
To develop a machine learning framework to forecast emergency department (ED) crowding and to evaluate model performance under spatial and temporal data drift. Materials and Methods We obtained 4 datasets, identified by the location: 1—large academic hospital and 2—rural hospital, and time period: pre-coronavirus disease (COVID) (January 1, 2019–February 1, 2020) and COVID-era (May 15, 2020–February 1, 2021). Our primary target was a binary outcome that is equal to 1 if the number of patients with acute respiratory illness that were ED boarding for more than 4 h was above a prescribed historical percentile. We trained a random forest and used the area under the curve (AUC) to evaluate out-of-sample performance for 2 experiments: (1) we evaluated the impact of sudden temporal drift by training models using pre-COVID data and testing them during the COVID-era, (2) we evaluated the impact of spatial drift by testing models trained at location 1 on data from location 2, and vice versa.
Results
The baseline AUC values for ED boarding ranged from 0.54 (pre-COVID at location 2) to 0.81 (COVID-era at location 1). Models trained with pre-COVID data performed similarly to COVID-era models (0.82 vs 0.78 at location 1). Models that were transferred from location 2 to location 1 performed worse than models trained at location 1 (0.51 vs 0.78). Discussion and Conclusion Our results demonstrate that ED boarding is a predictable metric for ED crowding, models were not significantly impacted by temporal data drift, and any attempts at implementation must consider spatial data drift.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Industrial and Systems Engineering, University of Wisconsin-Madison, Madison, Wisconsin, USA.
- BerbeeWalsh Department of Emergency Medicine, University of Wisconsin-Madison, Madison, Wisconsin, USA.
- Department of Biostatistics and Medical Informatics, University of Wisconsin-Madison, Madison, Wisconsin, USA.
- Marshfield Clinic Research Institute, Marshfield, Wisconsin, USA.
- Applied Data Science, Information Services, UW-Health, Madison, Wisconsin, USA.
ORCID-Profile: Ari J Smith, Brian W Patterson, Frank Liao, Justin J Boutilier
Förderung
- UW School of Medicine and Public Health
- the Wisconsin Partnership Program
Zitieren
APA
Smith, A. J., Patterson, B. W., Pulia, M. S., Mayer, J., Schwei, R. J., Nagarajan, R., Liao, F., Shah, M. N., & Boutilier, J. J. (2023). Multisite evaluation of prediction models for emergency department crowding before and during the COVID-19 pandemic. Journal of the American Medical Informatics Association. https://doi.org/10.1093/jamia/ocac214
Vancouver
Smith AJ, Patterson BW, Pulia MS, Mayer J, Schwei RJ, Nagarajan R, et al. Multisite evaluation of prediction models for emergency department crowding before and during the COVID-19 pandemic. Journal of the American Medical Informatics Association. 2023. doi:10.1093/jamia/ocac214
BibTeX
@article{Smith2023,
author = {Smith, Ari J and Patterson, Brian W and Pulia, Michael S and Mayer, John and Schwei, Rebecca J and Nagarajan, Radha and Liao, Frank and Shah, Manish N and Boutilier, Justin J},
title = {Multisite evaluation of prediction models for emergency department crowding before and during the COVID-19 pandemic},
journal = {Journal of the American Medical Informatics Association},
year = {2023},
doi = {10.1093/jamia/ocac214}
}
EUVIMED