EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC-ND 4.0

Disambiguation of acronyms in clinical narratives with large language models

Amila Kugic iD, Stefan Schulz, Markus Kreuzthaler iD

Journal of the American Medical Informatics Association · 2024

DOI: 10.1093/jamia/ocae157 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of the American Medical Informatics Association, 2024. Verfasst von Amila Kugic und 2 weiteren Autor:innen. Beteiligte Einrichtung: Institute for Medical Informatics, Statistics and Documentation, Medical University of Graz, 8036 Graz, Austria. Hauptthemen laut MEDLINE: Abbreviations as Topic, Natural Language Processing. Die Arbeit wurde bisher 17-mal zitiert (Crossref) und nennt 23 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0). Gefördert von: European Union's Horizon Research and Innovation Programme, European Union's Horizon Research and Innovation.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objective

To assess the performance of large language models (LLMs) for zero-shot disambiguation of acronyms in clinical narratives. Materials and Methods Clinical narratives in English, German, and Portuguese were applied for testing the performance of four LLMs: GPT-3.5, GPT-4, Llama-2-7b-chat, and Llama-2-70b-chat. For English, the anonymized Clinical Abbreviation Sense Inventory (CASI, University of Minnesota) was used. For German and Portuguese, at least 500 text spans were processed. The output of LLM models, prompted with contextual information, was analyzed to compare their acronym disambiguation capability, grouped by document-level metadata, the source language, and the LLM.

Results

On CASI, GPT-3.5 achieved 0.91 in accuracy. GPT-4 outperformed GPT-3.5 across all datasets, reaching 0.98 in accuracy for CASI, 0.86 and 0.65 for two German datasets, and 0.88 for Portuguese. Llama models only reached 0.73 for CASI and failed severely for German and Portuguese. Across LLMs, performance decreased from English to German and Portuguese processing languages. There was no evidence that additional document-level metadata had a significant effect.

Conclusion

For English clinical narratives, acronym resolution by GPT-4 can be recommended to improve readability of clinical text by patients and professionals. For German and Portuguese, better models are needed. Llama models, which are particularly interesting for processing sensitive content on premise, cannot yet be recommended for acronym resolution.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Abbreviations as TopicNatural Language ProcessingHumansLanguageNarrationElectronic Health Records

Stichwörter der Autor:innen

acronymselectronic health recordslarge language modelsnatural language processing

Autor:innen und Institutionen

  • Institute for Medical Informatics, Statistics and Documentation, Medical University of Graz, 8036 Graz, Austria.

ORCID-Profile: Amila Kugic, Markus Kreuzthaler

Förderung

  • European Union's Horizon Research and Innovation Programme
  • European Union's Horizon Research and Innovation

Zitieren

APA

Kugic, A., Schulz, S., & Kreuzthaler, M. (2024). Disambiguation of acronyms in clinical narratives with large language models. Journal of the American Medical Informatics Association. https://doi.org/10.1093/jamia/ocae157

Vancouver

Kugic A, Schulz S, Kreuzthaler M. Disambiguation of acronyms in clinical narratives with large language models. Journal of the American Medical Informatics Association. 2024. doi:10.1093/jamia/ocae157

BibTeX
@article{Kugic2024,
  author = {Kugic, Amila and Schulz, Stefan and Kreuzthaler, Markus},
  title = {Disambiguation of acronyms in clinical narratives with large language models},
  journal = {Journal of the American Medical Informatics Association},
  year = {2024},
  doi = {10.1093/jamia/ocae157}
}
Kostenlos starten →