EUVIMEDEUVIMED entdecken
Open Access

A comparative study of pretrained language models for long clinical text

Yikuan Li iD, Ramsey M Wehbe, Faraz S Ahmad iD, Hanyin Wang iD, Yuan Luo

Journal of the American Medical Informatics Association · 18.01.2023 · 30 (2) · 340-347

DOI: 10.1093/jamia/ocac225 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of the American Medical Informatics Association, online erschienen am 30.11.2022. Verfasst von Yikuan Li und 4 weiteren Autor:innen. Beteiligte Einrichtung: Division of Health and Biomedical Informatics, Department of Preventive Medicine, Feinberg School of Medicine, Northwestern University, Chicago, Illi…. Hauptthemen laut MEDLINE: Language, Learning. Die Arbeit wurde bisher 144-mal zitiert (Crossref) und nennt 46 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: NLM NIH HHS, NCATS NIH HHS, NIH HHS.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objective

Clinical knowledge-enriched transformer models (eg, ClinicalBERT) have state-of-the-art results on clinical natural language processing (NLP) tasks. One of the core limitations of these transformer models is the substantial memory consumption due to their full self-attention mechanism, which leads to the performance degradation in long clinical texts. To overcome this, we propose to leverage long-sequence transformer models (eg, Longformer and BigBird), which extend the maximum input sequence length from 512 to 4096, to enhance the ability to model long-term dependencies in long clinical texts. Materials and methods Inspired by the success of long-sequence transformer models and the fact that clinical notes are mostly long, we introduce 2 domain-enriched language models, Clinical-Longformer and Clinical-BigBird, which are pretrained on a large-scale clinical corpus. We evaluate both language models using 10 baseline tasks including named entity recognition, question answering, natural language inference, and document classification tasks.

Results

The results demonstrate that Clinical-Longformer and Clinical-BigBird consistently and significantly outperform ClinicalBERT and other short-sequence transformers in all 10 downstream tasks and achieve new state-of-the-art results.

Discussion

Our pretrained language models provide the bedrock for clinical NLP using long texts. We have made our source code available at https://github.com/luoyuanlab/Clinical-Longformer, and the pretrained models available for public download at: https://huggingface.co/yikuan8/Clinical-Longformer.

Conclusion

This study demonstrates that clinical knowledge-enriched long-sequence transformers are able to learn long-term dependencies in long clinical text. Our methods can also inspire the development of other domain-enriched long-sequence transformers.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

LanguageLearningNatural Language Processing

Stichwörter der Autor:innen

clinical natural language processingnamed entity recognitionnatural language inferencequestion answeringtext classification

Autor:innen und Institutionen

  • Division of Health and Biomedical Informatics, Department of Preventive Medicine, Feinberg School of Medicine, Northwestern University, Chicago, Illinois, USA.
  • Division of Cardiology, Department of Medicine, Feinberg School of Medicine, Northwestern University, Chicago, Illinois, USA.
  • Bluhm Cardiovascular Institute Center for Artificial Intelligence, Northwestern Medicine, Chicago, Illinois, USA.

ORCID-Profile: Yikuan Li, Faraz S Ahmad, Hanyin Wang

Förderung

  • NLM NIH HHS
  • NCATS NIH HHS
  • NIH HHS
  • National Institutes of Health

Zitieren

APA

Li, Y., Wehbe, R. M., Ahmad, F. S., Wang, H., & Luo, Y. (2023). A comparative study of pretrained language models for long clinical text. Journal of the American Medical Informatics Association, 30(2), 340-347. https://doi.org/10.1093/jamia/ocac225

Vancouver

Li Y, Wehbe RM, Ahmad FS, Wang H, Luo Y. A comparative study of pretrained language models for long clinical text. Journal of the American Medical Informatics Association. 2023;30(2):340-347. doi:10.1093/jamia/ocac225

BibTeX
@article{Li2023,
  author = {Li, Yikuan and Wehbe, Ramsey M and Ahmad, Faraz S and Wang, Hanyin and Luo, Yuan},
  title = {A comparative study of pretrained language models for long clinical text},
  journal = {Journal of the American Medical Informatics Association},
  year = {2023},
  volume = {30},
  number = {2},
  pages = {340--347},
  doi = {10.1093/jamia/ocac225}
}
Kostenlos starten →