EUVIMEDEUVIMED entdecken
ValidierungsstudieOpen Access

Measuring cognitive effort using tabular transformer-based language models of electronic health record-based audit log action sequences

Seunghwan Kim iD, Benjamin C Warner, Daphne Lew iD, Sunny S Lou iD, Thomas Kannampallil iD

Journal of the American Medical Informatics Association · 01.10.2024 · 31 (10) · 2228-2235

DOI: 10.1093/jamia/ocae171 ↗

Kurz gefasst

Validierungsstudie in der Zeitschrift Journal of the American Medical Informatics Association, online erschienen am 13.07.2024. Verfasst von Seunghwan Kim und 4 weiteren Autor:innen. Beteiligte Einrichtung: Roy and Diana Vagelos Division of Biology and Biomedical Sciences, Washington University in St. Louis, St. Louis, MO 63110, United States. Hauptthemen laut MEDLINE: Electronic Health Records, Cognition. Die Arbeit wurde bisher 8-mal zitiert (Crossref) und nennt 60 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: NHLBI NIH HHS, AHRQ HHS, Agency for Healthcare Research and Quality.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objectives

To develop and validate a novel measure, action entropy, for assessing the cognitive effort associated with electronic health record (EHR)-based work activities. Materials and Methods EHR-based audit logs of attending physicians and advanced practice providers (APPs) from four surgical intensive care units in 2019 were included. Neural language models (LMs) were trained and validated separately for attendings’ and APPs’ action sequences. Action entropy was calculated as the cross-entropy associated with the predicted probability of the next action, based on prior actions. To validate the measure, a matched pairs study was conducted to assess the difference in action entropy during known high cognitive effort scenarios, namely, attention switching between patients and to or from the EHR inbox.

Results

Sixty-five clinicians performing 5 904 429 EHR-based audit log actions on 8956 unique patients were included. All attention switching scenarios were associated with a higher action entropy compared to non-switching scenarios (P < .001), except for the from-inbox switching scenario among APPs. The highest difference among attendings was for the from-inbox attention switching: Action entropy was 1.288 (95% CI, 1.256-1.320) standard deviations (SDs) higher for switching compared to non-switching scenarios. For APPs, the highest difference was for the to-inbox switching, where action entropy was 2.354 (95% CI, 2.311-2.397) SDs higher for switching compared to non-switching scenarios.

Discussion

We developed a LM-based metric, action entropy, for assessing cognitive burden associated with EHR-based actions. The metric showed discriminant validity and statistical significance when evaluated against known situations of high cognitive effort (ie, attention switching). With additional validation, this metric can potentially be used as a screening tool for assessing behavioral action phenotypes that are associated with higher cognitive burden.

Conclusion

An LM-based action entropy metric—relying on sequences of EHR actions—offers opportunities for assessing cognitive effort in EHR-based workflows.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Electronic Health RecordsCognitionHumansMedical AuditMedical Staff, HospitalEntropyWorkload

Stichwörter der Autor:innen

audit logclinical workflowcognitive effortentropylanguage model

Autor:innen und Institutionen

  • Roy and Diana Vagelos Division of Biology and Biomedical Sciences, Washington University in St. Louis, St. Louis, MO 63110, United States.
  • Institute for Informatics, Data Science and Biostatistics (I2DB), Washington University School of Medicine, St. Louis, MO 63110, United States.
  • Department of Computer Science and Engineering, Washington University St. Louis, St. Louis, MO 63130-4899, United States.
  • Department of Anesthesiology, Washington University School of Medicine, St. Louis, MO 63110, United States.

ORCID-Profile: Seunghwan Kim, Daphne Lew, Sunny S Lou, Thomas Kannampallil

Förderung

  • NHLBI NIH HHS
  • AHRQ HHS
  • Agency for Healthcare Research and Quality

Zitieren

APA

Kim, S., Warner, B. C., Lew, D., Lou, S. S., & Kannampallil, T. (2024). Measuring cognitive effort using tabular transformer-based language models of electronic health record-based audit log action sequences. Journal of the American Medical Informatics Association, 31(10), 2228-2235. https://doi.org/10.1093/jamia/ocae171

Vancouver

Kim S, Warner BC, Lew D, Lou SS, Kannampallil T. Measuring cognitive effort using tabular transformer-based language models of electronic health record-based audit log action sequences. Journal of the American Medical Informatics Association. 2024;31(10):2228-2235. doi:10.1093/jamia/ocae171

BibTeX
@article{Kim2024,
  author = {Kim, Seunghwan and Warner, Benjamin C and Lew, Daphne and Lou, Sunny S and Kannampallil, Thomas},
  title = {Measuring cognitive effort using tabular transformer-based language models of electronic health record-based audit log action sequences},
  journal = {Journal of the American Medical Informatics Association},
  year = {2024},
  volume = {31},
  number = {10},
  pages = {2228--2235},
  doi = {10.1093/jamia/ocae171}
}
Kostenlos starten →