Evaluation of a Digital Scribe: Conversation Summarization for Emergency Department Consultation Calls
Applied Clinical Informatics · 01.05.2024 · 15 (03) · 600-611
DOI: 10.1055/a-2327-4121 ↗Kurz gefasst
Fachartikel in der Zeitschrift Applied Clinical Informatics, erschienen am 01.05.2024. Verfasst von Emre Sezgin und 2 weiteren Autor:innen. Beteiligte Einrichtung: Nationwide Children's Hospital, Columbus, United States. Die Arbeit wurde bisher 23-mal zitiert (Crossref) und nennt 47 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: NCATS NIH HHS, U.S. Department of Health and Human Services.
Abstract (Originalsprache)
Objectives
We present a proof-of-concept digital scribe system as an emergency department (ED) consultation call-based clinical conversation summarization pipeline to support clinical documentation and report its performance.
Methods
We use four pretrained large language models to establish the digital scribe system: T5-small, T5-base, PEGASUS-PubMed, and BART-Large-CNN via zero-shot and fine-tuning approaches. Our dataset includes 100 referral conversations among ED clinicians and medical records. We report the ROUGE-1, ROUGE-2, and ROUGE-L to compare model performance. In addition, we annotated transcriptions to assess the quality of generated summaries.
Results
The fine-tuned BART-Large-CNN model demonstrates greater performance in summarization tasks with the highest ROUGE scores (F1ROUGE-1 = 0.49, F1ROUGE-2 = 0.23, F1ROUGE-L = 0.35) scores. In contrast, PEGASUS-PubMed lags notably (F1ROUGE-1 = 0.28, F1ROUGE-2 = 0.11, F1ROUGE-L = 0.22). BART-Large-CNN's performance decreases by more than 50% with the zero-shot approach. Annotations show that BART-Large-CNN performs 71.4% recall in identifying key information and a 67.7% accuracy rate.
Conclusion
The BART-Large-CNN model demonstrates a high level of understanding of clinical dialogue structure, indicated by its performance with and without fine-tuning. Despite some instances of high recall, there is variability in the model's performance, particularly in achieving consistent correctness, suggesting room for refinement. The model's recall ability varies across different information categories. The study provides evidence toward the potential of artificial intelligence-assisted tools in assisting clinical documentation. Future work is suggested on expanding the research scope with additional language models and hybrid approaches and comparative analysis to measure documentation burden and human factors.
Autor:innen und Institutionen
- Nationwide Children's Hospital, Columbus, United States.
- IT Research Innovation - Data Science, Nationwide Children's Hospital, Columbus, United States.
Förderung
- NCATS NIH HHS
- U.S. Department of Health and Human Services
Zitieren
APA
Sezgin, E., Sirrianni, J. W., & Kranz, K. (2024). Evaluation of a Digital Scribe: Conversation Summarization for Emergency Department Consultation Calls. Applied Clinical Informatics, 15(03), 600-611. https://doi.org/10.1055/a-2327-4121
Vancouver
Sezgin E, Sirrianni JW, Kranz K. Evaluation of a Digital Scribe: Conversation Summarization for Emergency Department Consultation Calls. Applied Clinical Informatics. 2024;15(03):600-611. doi:10.1055/a-2327-4121
BibTeX
@article{Sezgin2024,
author = {Sezgin, Emre and Sirrianni, Joseph Winstead and Kranz, Kelly},
title = {Evaluation of a Digital Scribe: Conversation Summarization for Emergency Department Consultation Calls},
journal = {Applied Clinical Informatics},
year = {2024},
volume = {15},
number = {03},
pages = {600--611},
doi = {10.1055/a-2327-4121}
}
EUVIMED