On the effectiveness of compact biomedical transformers
Bioinformatics · 01.03.2023 · 39 (3)
DOI: 10.1093/bioinformatics/btad103 ↗Kurz gefasst
Fachartikel in der Zeitschrift Bioinformatics, online erschienen am 24.02.2023. Verfasst von Omid Rohanian und 3 weiteren Autor:innen. Beteiligte Einrichtung: Department of Engineering Science, University of Oxford, Oxford, UK. Hauptthemen laut MEDLINE: Natural Language Processing. Die Arbeit wurde bisher 41-mal zitiert (Crossref) und nennt 28 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Institute for Health Research.
Abstract (Originalsprache)
Motivation Language models pre-trained on biomedical corpora, such as BioBERT, have recently shown promising results on downstream biomedical tasks. Many existing pre-trained models, on the other hand, are resource-intensive and computationally heavy owing to factors such as embedding size, hidden dimension and number of layers. The natural language processing community has developed numerous strategies to compress these models utilizing techniques such as pruning, quantization and knowledge distillation, resulting in models that are considerably faster, smaller and subsequently easier to use in practice. By the same token, in this article, we introduce six lightweight models, namely, BioDistilBERT, BioTinyBERT, BioMobileBERT, DistilBioBERT, TinyBioBERT and CompactBioBERT which are obtained either by knowledge distillation from a biomedical teacher or continual learning on the Pubmed dataset. We evaluate all of our models on three biomedical tasks and compare them with BioBERT-v1.1 to create the best efficient lightweight models that perform on par with their larger counterparts. Results We trained six different models in total, with the largest model having 65 million in parameters and the smallest having 15 million; a far lower range of parameters compared with BioBERT’s 110M. Based on our experiments on three different biomedical tasks, we found that models distilled from a biomedical teacher and models that have been additionally pre-trained on the PubMed dataset can retain up to 98.8% and 98.6% of the performance of the BioBERT-v1.1, respectively. Overall, our best model below 30 M parameters is BioMobileBERT, while our best models over 30 M parameters are DistilBioBERT and CompactBioBERT, which can keep up to 98.2% and 98.8% of the performance of the BioBERT-v1.1, respectively. Availability and implementation Codes are available at: https://github.com/nlpie-research/Compact-Biomedical-Transformers. Trained models can be accessed at: https://huggingface.co/nlpie.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Autor:innen und Institutionen
- Department of Engineering Science, University of Oxford, Oxford, UK.
- NLPie Research, Oxford, UK.
- Department of Electrical and Electronic Engineering, University of Surrey, Guildford, UK.
- Oxford-Suzhou Centre for Advanced Research, Suzhou, China.
ORCID-Profile: Omid Rohanian
Förderung
- National Institute for Health Research
Zitieren
APA
Rohanian, O., Nouriborji, M., Kouchaki, S., & Clifton, D. A. (2023). On the effectiveness of compact biomedical transformers. Bioinformatics, 39(3). https://doi.org/10.1093/bioinformatics/btad103
Vancouver
Rohanian O, Nouriborji M, Kouchaki S, Clifton DA. On the effectiveness of compact biomedical transformers. Bioinformatics. 2023;39(3). doi:10.1093/bioinformatics/btad103
BibTeX
@article{Rohanian2023,
author = {Rohanian, Omid and Nouriborji, Mohammadmahdi and Kouchaki, Samaneh and Clifton, David A},
title = {On the effectiveness of compact biomedical transformers},
journal = {Bioinformatics},
year = {2023},
volume = {39},
number = {3},
doi = {10.1093/bioinformatics/btad103}
}
EUVIMED