EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Can GPT-3.5 generate and code discharge summaries?

Matúš Falis iD, Aryo Pradipta Gema iD, Hang Dong iD, Luke Daines iD, Siddharth Basetti iD, Michael Holder iD, Rose S Penfold iD, Alexandra Birch iD, Beatrice Alex iD

Journal of the American Medical Informatics Association · 2024

DOI: 10.1093/jamia/ocae132 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of the American Medical Informatics Association, 2024. Verfasst von Matúš Falis und 8 weiteren Autor:innen. Beteiligte Einrichtung: School of Informatics, The University of Edinburgh, Edinburgh EH8 9AB, United Kingdom. Hauptthemen laut MEDLINE: International Classification of Diseases, Clinical Coding, Patient Discharge Summaries. Die Arbeit wurde bisher 22-mal zitiert (Crossref) und nennt 34 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Wellcome Trust, UKRI, National Institute for Health Research.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objectives

The aim of this study was to investigate GPT-3.5 in generating and coding medical documents with International Classification of Diseases (ICD)-10 codes for data augmentation on low-resource labels. Materials and Methods Employing GPT-3.5 we generated and coded 9606 discharge summaries based on lists of ICD-10 code descriptions of patients with infrequent (or generation) codes within the MIMIC-IV dataset. Combined with the baseline training set, this formed an augmented training set. Neural coding models were trained on baseline and augmented data and evaluated on an MIMIC-IV test set. We report micro- and macro-F1 scores on the full codeset, generation codes, and their families. Weak Hierarchical Confusion Matrices determined within-family and outside-of-family coding errors in the latter codesets. The coding performance of GPT-3.5 was evaluated on prompt-guided self-generated data and real MIMIC-IV data. Clinicians evaluated the clinical acceptability of the generated documents.

Results

Data augmentation results in slightly lower overall model performance but improves performance for the generation candidate codes and their families, including 1 absent from the baseline training data. Augmented models display lower out-of-family error rates. GPT-3.5 identifies ICD-10 codes by their prompted descriptions but underperforms on real data. Evaluators highlight the correctness of generated concepts while suffering in variety, supporting information, and narrative. Discussion and Conclusion While GPT-3.5 alone given our prompt setting is unsuitable for ICD-10 coding, it supports data augmentation for training neural models. Augmentation positively affects generation code families but mainly benefits codes with existing examples. Augmentation reduces out-of-family errors. Documents generated by GPT-3.5 state prompted concepts correctly but lack variety, and authenticity in narratives.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

International Classification of DiseasesClinical CodingPatient Discharge SummariesHumansElectronic Health RecordsPatient DischargeNeural Networks, Computer

Stichwörter der Autor:innen

ICD codingclinical text generationdata augmentationevaluation by clinicianslarge language model

Autor:innen und Institutionen

  • School of Informatics, The University of Edinburgh, Edinburgh EH8 9AB, United Kingdom.
  • Department of Computer Science, University of Exeter, Exeter EX4 4QF, United Kingdom.
  • Centre for Medical Informatics, Usher Institute, University of Edinburgh, Edinburgh EH16 4UX, United Kingdom.
  • Department of Research, Development and Innovation, National Health Service Highland, Inverness IV2 3JH, United Kingdom.
  • Centre for Population Health Sciences, Usher Institute, The University of Edinburgh, Edinburgh EH16 4UX, United Kingdom.
  • Ageing and Health, Usher Institute, The University of Edinburgh, Edinburgh EH16 4UX, United Kingdom.

ORCID-Profile: Matúš Falis, Aryo Pradipta Gema, Hang Dong, Luke Daines, Siddharth Basetti, Michael Holder, Rose S Penfold, Alexandra Birch, Beatrice Alex

Förderung

  • Wellcome Trust
  • UKRI
  • National Institute for Health Research
  • Artificial Intelligence and Multimorbidity: Clustering in Individuals, Space and Clinical Context
  • United Kingdom Research and Innovation
  • Multimorbidity Doctoral Training Programme for Health Professionals
  • Legal and General PLC
  • Advanced Care Research Centre

Zitieren

APA

Falis, M., Gema, A. P., Dong, H., Daines, L., Basetti, S., Holder, M., Penfold, R. S., Birch, A., & Alex, B. (2024). Can GPT-3.5 generate and code discharge summaries?. Journal of the American Medical Informatics Association. https://doi.org/10.1093/jamia/ocae132

Vancouver

Falis M, Gema AP, Dong H, Daines L, Basetti S, Holder M, et al. Can GPT-3.5 generate and code discharge summaries?. Journal of the American Medical Informatics Association. 2024. doi:10.1093/jamia/ocae132

BibTeX
@article{Falis2024,
  author = {Falis, Matúš and Gema, Aryo Pradipta and Dong, Hang and Daines, Luke and Basetti, Siddharth and Holder, Michael and Penfold, Rose S and Birch, Alexandra and Alex, Beatrice},
  title = {Can GPT-3.5 generate and code discharge summaries?},
  journal = {Journal of the American Medical Informatics Association},
  year = {2024},
  doi = {10.1093/jamia/ocae132}
}
Kostenlos starten →