From Revisions to Insights: Converting Radiology Report Revisions into Actionable Educational Feedback Using Generative AI Models
Journal of Imaging Informatics in Medicine · 19.08.2024 · 38 (2) · 1265-1279
DOI: 10.1007/s10278-024-01233-4 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Imaging Informatics in Medicine, online erschienen am 19.08.2024. Verfasst von Shawn Lyo und 5 weiteren Autor:innen. Beteiligte Einrichtung: Department of Radiology, Hospital of the University of Pennsylvania, Philadelphia, PA, USA. shawn.kt.lyo@gmail.com. Hauptthemen laut MEDLINE: Radiology, Formative Feedback. Die Arbeit wurde bisher 25-mal zitiert (Crossref) und nennt 31 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Expert feedback on trainees’ preliminary reports is crucial for radiologic training, but real-time feedback can be challenging due to non-contemporaneous, remote reading and increasing imaging volumes. Trainee report revisions contain valuable educational feedback, but synthesizing data from raw revisions is challenging. Generative AI models can potentially analyze these revisions and provide structured, actionable feedback. This study used the OpenAI GPT-4 Turbo API to analyze paired synthesized and open-source analogs of preliminary and finalized reports, identify discrepancies, categorize their severity and type, and suggest review topics. Expert radiologists reviewed the output by grading discrepancies, evaluating the severity and category accuracy, and suggested review topic relevance. The reproducibility of discrepancy detection and maximal discrepancy severity was also examined. The model exhibited high sensitivity, detecting significantly more discrepancies than radiologists ( W = 19.0, p < 0.001) with a strong positive correlation ( r = 0.778, p < 0.001). Interrater reliability for severity and type were fair (Fleiss’ kappa = 0.346 and 0.340, respectively; weighted kappa = 0.622 for severity). The LLM achieved a weighted F1 score of 0.66 for severity and 0.64 for type. Generated teaching points were considered relevant in ~ 85% of cases, and relevance correlated with the maximal discrepancy severity (Spearman ρ = 0.76, p < 0.001). The reproducibility was moderate to good (ICC (2,1) = 0.690) for the number of discrepancies and substantial for maximal discrepancy severity (Fleiss’ kappa = 0.718; weighted kappa = 0.94). Generative AI models can effectively identify discrepancies in report revisions and generate relevant educational feedback, offering promise for enhancing radiology training.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Radiology, Hospital of the University of Pennsylvania, Philadelphia, PA, USA. shawn.kt.lyo@gmail.com.
- Department of Radiology, Hospital of the University of Pennsylvania, Philadelphia, PA, USA.
Zitieren
APA
Lyo, S., Mohan, S., Hassankhani, A., Noor, A., Dako, F., & Cook, T. (2024). From Revisions to Insights: Converting Radiology Report Revisions into Actionable Educational Feedback Using Generative AI Models. Journal of Imaging Informatics in Medicine, 38(2), 1265-1279. https://doi.org/10.1007/s10278-024-01233-4
Vancouver
Lyo S, Mohan S, Hassankhani A, Noor A, Dako F, Cook T. From Revisions to Insights: Converting Radiology Report Revisions into Actionable Educational Feedback Using Generative AI Models. Journal of Imaging Informatics in Medicine. 2024;38(2):1265-1279. doi:10.1007/s10278-024-01233-4
BibTeX
@article{Lyo2024,
author = {Lyo, Shawn and Mohan, Suyash and Hassankhani, Alvand and Noor, Abass and Dako, Farouk and Cook, Tessa},
title = {From Revisions to Insights: Converting Radiology Report Revisions into Actionable Educational Feedback Using Generative AI Models},
journal = {Journal of Imaging Informatics in Medicine},
year = {2024},
volume = {38},
number = {2},
pages = {1265--1279},
doi = {10.1007/s10278-024-01233-4}
}
EUVIMED