EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Pretraining effective T5 generative models for clinical and biomedical applications

Saad Althabiti iD, Chuming Chen, Sultan Alrowili, Cathy Wu iD, K Vijay-Shanker

PLOS One · 17.04.2026 · 21 (4) · e0342610

DOI: 10.1371/journal.pone.0342610 ↗

Kurz gefasst

Fachartikel in der Zeitschrift PLOS One, online erschienen am 17.04.2026. Verfasst von Saad Althabiti und 4 weiteren Autor:innen. Beteiligte Einrichtung: Department of Health Informatics, King Saud bin Abdulaziz University for Health Sciences, Riyadh, Saudi Arabia. Hauptthemen laut MEDLINE: Natural Language Processing. Die Arbeit nennt 30 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Saudi Arabian Cultural Mission, National Institutes of Health, Delaware IDeA Network of Biomedical Research Excellence.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

This paper presents a study of the impact of corpus selection and vocabulary design on the performance of T5-based language models in clinical and biomedical domains. We introduce five different T5-EHR models, each pretrained from scratch using different combinations of clinical and biomedical corpora alongside domain-specific vocabularies. We evaluated these models across a variety of clinical and biomedical tasks to quantify the impact of pretraining data and vocabulary tokenization choices on downstream performance. Our findings reveal the importance of aligning both pretraining corpus and vocabulary with the target domain. Models pretrained exclusively on clinical data achieve superior performance on clinical tasks, while adding biomedical data contributes only marginal gains in most cases, with a few exceptions. Similarly, the choice of vocabulary significantly influences model performance, with clinical-specific vocabularies outperforming general biomedical vocabularies in tasks requiring a deeper understanding of clinical language. Also, the T5 generative models perform competitively with state-of-the-art discriminative models on several biomedical benchmarks, demonstrating strong generalization to biomedical domain. Overall, these results emphasize that task-specific selection of corpus and vocabulary is essential for optimizing model performance in clinical and biomedical natural language processing (NLP).

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Natural Language ProcessingHumansGenerative Artificial IntelligenceVocabularyLarge Language ModelsVocabulary, Controlled

Autor:innen und Institutionen

  • Department of Health Informatics, King Saud bin Abdulaziz University for Health Sciences, Riyadh, Saudi Arabia.
  • King Abdullah International Medical Research Center, Riyadh, Saudi Arabia.
  • Ministry of the National Guard - Health Affairs, Riyadh, Saudi Arabia.
  • Center for Bioinformatics and Computational Biology, University of Delaware, Newark, Delaware, United States of America.
  • Department of Computer & Information Sciences, University of Delaware, Newark, Delaware, United States of America.
  • IBM Research, Riyadh, Saudi Arabia.

ORCID-Profile: Saad Althabiti, Cathy Wu

Förderung

  • Saudi Arabian Cultural Mission
  • National Institutes of Health
  • Delaware IDeA Network of Biomedical Research Excellence

Zitieren

APA

Althabiti, S., Chen, C., Alrowili, S., Wu, C., & Vijay-Shanker, K. (2026). Pretraining effective T5 generative models for clinical and biomedical applications. PLOS One, 21(4), e0342610. https://doi.org/10.1371/journal.pone.0342610

Vancouver

Althabiti S, Chen C, Alrowili S, Wu C, Vijay-Shanker K. Pretraining effective T5 generative models for clinical and biomedical applications. PLOS One. 2026;21(4):e0342610. doi:10.1371/journal.pone.0342610

BibTeX
@article{Althabiti2026,
  author = {Althabiti, Saad and Chen, Chuming and Alrowili, Sultan and Wu, Cathy and Vijay-Shanker, K},
  title = {Pretraining effective T5 generative models for clinical and biomedical applications},
  journal = {PLOS One},
  year = {2026},
  volume = {21},
  number = {4},
  pages = {e0342610},
  doi = {10.1371/journal.pone.0342610}
}
Kostenlos starten →