Automatic structuring of radiology reports with on-premise open-source large language models
European Radiology · 10.10.2024 · 35 (4) · 2018-2029
DOI: 10.1007/s00330-024-11074-y ↗Kurz gefasst
Fachartikel in der Zeitschrift European Radiology, online erschienen am 10.10.2024. Verfasst von Piotr Woźnicki und 11 weiteren Autor:innen. Beteiligte Einrichtung: Department of Diagnostic and Interventional Radiology, University Hospital Würzburg, Würzburg, Germany. woznicki_p@ukw.de. Hauptthemen laut MEDLINE: Radiology Information Systems, Radiography, Thoracic. Die Arbeit wurde bisher 52-mal zitiert (Crossref) und nennt 24 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Bundesministerium für Bildung und Forschung.
Abstract (Originalsprache)
Objectives
Structured reporting enhances comparability, readability, and content detail. Large language models (LLMs) could convert free text into structured data without disrupting radiologists’ reporting workflow. This study evaluated an on-premise, privacy-preserving LLM for automatically structuring free-text radiology reports. Materials and methods We developed an approach to controlling the LLM output, ensuring the validity and completeness of structured reports produced by a locally hosted Llama-2-70B-chat model. A dataset with de-identified narrative chest radiograph (CXR) reports was compiled retrospectively. It included 202 English reports from a publicly available MIMIC-CXR dataset and 197 German reports from our university hospital. Senior radiologist prepared a detailed, fully structured reporting template with 48 question-answer pairs. All reports were independently structured by the LLM and two human readers. Bayesian inference (Markov chain Monte Carlo sampling) was used to estimate the distributions of Matthews correlation coefficient (MCC), with [−0.05, 0.05] as the region of practical equivalence (ROPE).
Results
The LLM generated valid structured reports in all cases, achieving an average MCC of 0.75 (94% HDI: 0.70–0.80) and F1 score of 0.70 (0.70–0.80) for English, and 0.66 (0.62–0.70) and 0.68 (0.64–0.72) for German reports, respectively. The MCC differences between LLM and humans were within ROPE for both languages: 0.01 (−0.05 to 0.07), 0.01 (−0.05 to 0.07) for English, and −0.01 (−0.07 to 0.05), 0.00 (−0.06 to 0.06) for German, indicating approximately comparable performance.
Conclusion
Locally hosted, open-source LLMs can automatically structure free-text radiology reports with approximately human accuracy. However, the understanding of semantics varied across languages and imaging findings. Key Points Question Why has structured reporting not been widely adopted in radiology despite clear benefits and how can we improve this?
Findings
A locally hosted large language model successfully structured narrative reports, showing variation between languages and findings. Critical relevance Structured reporting provides many benefits, but its integration into the clinical routine is limited. Automating the extraction of structured information from radiology reports enables the capture of structured data while allowing the radiologist to maintain their reporting workflow.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Diagnostic and Interventional Radiology, University Hospital Würzburg, Würzburg, Germany. woznicki_p@ukw.de.
- Department of Diagnostic and Interventional Radiology, University Hospital Würzburg, Würzburg, Germany.
- Department of Diagnostic and Interventional Radiology, University Hospital Aachen, Aachen, Germany.
- Department of Internal Medicine III, Heidelberg University Hospital, Heidelberg, Germany.
- DZHK (German Centre for Cardiovascular Research), Partner Site Heidelberg/Mannheim, Heidelberg, Germany.
- Department of Internal Medicine I, University Hospital Carl Gustav Carus, Technical University Dresden, Dresden, Germany.
Förderung
- Bundesministerium für Bildung und Forschung
Zitieren
APA
Woźnicki, P., Laqua, C., Fiku, I., Hekalo, A., Truhn, D., Engelhardt, S., Kather, J., Foersch, S., D'Antonoli, T. A., Pinto Dos Santos, D., Baeßler, B., & Laqua, F. C. (2024). Automatic structuring of radiology reports with on-premise open-source large language models. European Radiology, 35(4), 2018-2029. https://doi.org/10.1007/s00330-024-11074-y
Vancouver
Woźnicki P, Laqua C, Fiku I, Hekalo A, Truhn D, Engelhardt S, et al. Automatic structuring of radiology reports with on-premise open-source large language models. European Radiology. 2024;35(4):2018-2029. doi:10.1007/s00330-024-11074-y
BibTeX
@article{Wonicki2024,
author = {Woźnicki, Piotr and Laqua, Caroline and Fiku, Ina and Hekalo, Amar and Truhn, Daniel and Engelhardt, Sandy and Kather, Jakob and Foersch, Sebastian and D'Antonoli, Tugba Akinci and Pinto Dos Santos, Daniel and Baeßler, Bettina and Laqua, Fabian Christopher},
title = {Automatic structuring of radiology reports with on-premise open-source large language models},
journal = {European Radiology},
year = {2024},
volume = {35},
number = {4},
pages = {2018--2029},
doi = {10.1007/s00330-024-11074-y}
}
EUVIMED