Can GPT-3.5 generate and code discharge summaries?
Journal of the American Medical Informatics Association · 2024
DOI: 10.1093/jamia/ocae132 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of the American Medical Informatics Association, 2024. Verfasst von Matúš Falis und 8 weiteren Autor:innen. Beteiligte Einrichtung: School of Informatics, The University of Edinburgh, Edinburgh EH8 9AB, United Kingdom. Hauptthemen laut MEDLINE: International Classification of Diseases, Clinical Coding, Patient Discharge Summaries. Die Arbeit wurde bisher 22-mal zitiert (Crossref) und nennt 34 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Wellcome Trust, UKRI, National Institute for Health Research.
Abstract (Originalsprache)
Objectives
The aim of this study was to investigate GPT-3.5 in generating and coding medical documents with International Classification of Diseases (ICD)-10 codes for data augmentation on low-resource labels. Materials and Methods Employing GPT-3.5 we generated and coded 9606 discharge summaries based on lists of ICD-10 code descriptions of patients with infrequent (or generation) codes within the MIMIC-IV dataset. Combined with the baseline training set, this formed an augmented training set. Neural coding models were trained on baseline and augmented data and evaluated on an MIMIC-IV test set. We report micro- and macro-F1 scores on the full codeset, generation codes, and their families. Weak Hierarchical Confusion Matrices determined within-family and outside-of-family coding errors in the latter codesets. The coding performance of GPT-3.5 was evaluated on prompt-guided self-generated data and real MIMIC-IV data. Clinicians evaluated the clinical acceptability of the generated documents.
Results
Data augmentation results in slightly lower overall model performance but improves performance for the generation candidate codes and their families, including 1 absent from the baseline training data. Augmented models display lower out-of-family error rates. GPT-3.5 identifies ICD-10 codes by their prompted descriptions but underperforms on real data. Evaluators highlight the correctness of generated concepts while suffering in variety, supporting information, and narrative. Discussion and Conclusion While GPT-3.5 alone given our prompt setting is unsuitable for ICD-10 coding, it supports data augmentation for training neural models. Augmentation positively affects generation code families but mainly benefits codes with existing examples. Augmentation reduces out-of-family errors. Documents generated by GPT-3.5 state prompted concepts correctly but lack variety, and authenticity in narratives.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- School of Informatics, The University of Edinburgh, Edinburgh EH8 9AB, United Kingdom.
- Department of Computer Science, University of Exeter, Exeter EX4 4QF, United Kingdom.
- Centre for Medical Informatics, Usher Institute, University of Edinburgh, Edinburgh EH16 4UX, United Kingdom.
- Department of Research, Development and Innovation, National Health Service Highland, Inverness IV2 3JH, United Kingdom.
- Centre for Population Health Sciences, Usher Institute, The University of Edinburgh, Edinburgh EH16 4UX, United Kingdom.
- Ageing and Health, Usher Institute, The University of Edinburgh, Edinburgh EH16 4UX, United Kingdom.
ORCID-Profile: Matúš Falis, Aryo Pradipta Gema, Hang Dong, Luke Daines, Siddharth Basetti, Michael Holder, Rose S Penfold, Alexandra Birch, Beatrice Alex
Förderung
- Wellcome Trust
- UKRI
- National Institute for Health Research
- Artificial Intelligence and Multimorbidity: Clustering in Individuals, Space and Clinical Context
- United Kingdom Research and Innovation
- Multimorbidity Doctoral Training Programme for Health Professionals
- Legal and General PLC
- Advanced Care Research Centre
Zitieren
APA
Falis, M., Gema, A. P., Dong, H., Daines, L., Basetti, S., Holder, M., Penfold, R. S., Birch, A., & Alex, B. (2024). Can GPT-3.5 generate and code discharge summaries?. Journal of the American Medical Informatics Association. https://doi.org/10.1093/jamia/ocae132
Vancouver
Falis M, Gema AP, Dong H, Daines L, Basetti S, Holder M, et al. Can GPT-3.5 generate and code discharge summaries?. Journal of the American Medical Informatics Association. 2024. doi:10.1093/jamia/ocae132
BibTeX
@article{Falis2024,
author = {Falis, Matúš and Gema, Aryo Pradipta and Dong, Hang and Daines, Luke and Basetti, Siddharth and Holder, Michael and Penfold, Rose S and Birch, Alexandra and Alex, Beatrice},
title = {Can GPT-3.5 generate and code discharge summaries?},
journal = {Journal of the American Medical Informatics Association},
year = {2024},
doi = {10.1093/jamia/ocae132}
}
EUVIMED