Pretraining effective T5 generative models for clinical and biomedical applications
PLOS One · 17.04.2026 · 21 (4) · e0342610
DOI: 10.1371/journal.pone.0342610 ↗Kurz gefasst
Fachartikel in der Zeitschrift PLOS One, online erschienen am 17.04.2026. Verfasst von Saad Althabiti und 4 weiteren Autor:innen. Beteiligte Einrichtung: Department of Health Informatics, King Saud bin Abdulaziz University for Health Sciences, Riyadh, Saudi Arabia. Hauptthemen laut MEDLINE: Natural Language Processing. Die Arbeit nennt 30 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Saudi Arabian Cultural Mission, National Institutes of Health, Delaware IDeA Network of Biomedical Research Excellence.
Abstract (Originalsprache)
This paper presents a study of the impact of corpus selection and vocabulary design on the performance of T5-based language models in clinical and biomedical domains. We introduce five different T5-EHR models, each pretrained from scratch using different combinations of clinical and biomedical corpora alongside domain-specific vocabularies. We evaluated these models across a variety of clinical and biomedical tasks to quantify the impact of pretraining data and vocabulary tokenization choices on downstream performance. Our findings reveal the importance of aligning both pretraining corpus and vocabulary with the target domain. Models pretrained exclusively on clinical data achieve superior performance on clinical tasks, while adding biomedical data contributes only marginal gains in most cases, with a few exceptions. Similarly, the choice of vocabulary significantly influences model performance, with clinical-specific vocabularies outperforming general biomedical vocabularies in tasks requiring a deeper understanding of clinical language. Also, the T5 generative models perform competitively with state-of-the-art discriminative models on several biomedical benchmarks, demonstrating strong generalization to biomedical domain. Overall, these results emphasize that task-specific selection of corpus and vocabulary is essential for optimizing model performance in clinical and biomedical natural language processing (NLP).
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Autor:innen und Institutionen
- Department of Health Informatics, King Saud bin Abdulaziz University for Health Sciences, Riyadh, Saudi Arabia.
- King Abdullah International Medical Research Center, Riyadh, Saudi Arabia.
- Ministry of the National Guard - Health Affairs, Riyadh, Saudi Arabia.
- Center for Bioinformatics and Computational Biology, University of Delaware, Newark, Delaware, United States of America.
- Department of Computer & Information Sciences, University of Delaware, Newark, Delaware, United States of America.
- IBM Research, Riyadh, Saudi Arabia.
ORCID-Profile: Saad Althabiti, Cathy Wu
Förderung
- Saudi Arabian Cultural Mission
- National Institutes of Health
- Delaware IDeA Network of Biomedical Research Excellence
Zitieren
APA
Althabiti, S., Chen, C., Alrowili, S., Wu, C., & Vijay-Shanker, K. (2026). Pretraining effective T5 generative models for clinical and biomedical applications. PLOS One, 21(4), e0342610. https://doi.org/10.1371/journal.pone.0342610
Vancouver
Althabiti S, Chen C, Alrowili S, Wu C, Vijay-Shanker K. Pretraining effective T5 generative models for clinical and biomedical applications. PLOS One. 2026;21(4):e0342610. doi:10.1371/journal.pone.0342610
BibTeX
@article{Althabiti2026,
author = {Althabiti, Saad and Chen, Chuming and Alrowili, Sultan and Wu, Cathy and Vijay-Shanker, K},
title = {Pretraining effective T5 generative models for clinical and biomedical applications},
journal = {PLOS One},
year = {2026},
volume = {21},
number = {4},
pages = {e0342610},
doi = {10.1371/journal.pone.0342610}
}
EUVIMED