The impact of tokenizer selection in genomic language models
Bioinformatics · 2025
DOI: 10.1093/bioinformatics/btaf456 ↗Kurz gefasst
Fachartikel in der Zeitschrift Bioinformatics, 2025. Verfasst von LeAnn M Lindsey und 7 weiteren Autor:innen. Beteiligte Einrichtung: Kahlert School of Computing, University of Utah, Salt Lake City, Utah 84112, United States. Hauptthemen laut MEDLINE: Natural Language Processing, Genomics, Models, Genetic. Die Arbeit wurde bisher 12-mal zitiert (Crossref) und nennt 34 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Science Foundation, NCATS NIH HHS, NIH HPC Biowulf cluster.
Abstract (Originalsprache)
Motivation Genomic language models have recently emerged as a new method to decode, interpret, and generate genetic sequences. Existing genomic language models have utilized various tokenization methods, including character tokenization, overlapping and nonoverlapping k-mer tokenization, and byte-pair encoding, a method widely used in natural language models. Genomic sequences differ from natural language because of their low character variability, complex and overlapping features, and inconsistent directionality. These features make subword tokenization in genomic language models significantly different from both traditional language models and protein language models. Results This study explores the impact of tokenization in genomic language models by evaluating their downstream performance on 44 classification fine-tuning tasks. We also perform a direct comparison of byte pair encoding and character tokenization in Mamba, a state-space model. Our results indicate that character tokenization outperforms subword tokenization methods on tasks that rely on nucleotide-level resolution, such as splice site prediction and promoter detection. While byte-pair tokenization had stronger performance on the SARS-CoV-2 variant classification task, we observed limited statistically significant differences between tokenization methods on the remaining downstream tasks. Availability and implementation Detailed results of all benchmarking experiments are available in https://github.com/leannmlindsey/DNAtokenization. Training datasets and pretrained models are available at https://huggingface.co/datasets/leannmlindsey. Datasets and processing scripts are available at doi: 10.5281/zenodo.16287401 and doi: 10.5281/zenodo.16287130.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Autor:innen und Institutionen
- Kahlert School of Computing, University of Utah, Salt Lake City, Utah 84112, United States.
- National Library of Medicine, National Institutes of Health, Bethesda, Maryland 20892, United States.
- Department of Pediatrics, School of Medicine, University of Utah, Salt Lake City, Utah 84112, United States.
- Department of Pathology, School of Medicine, University of Utah, Salt Lake City, Utah 84112, United States.
- Department of Computer Science, Tufts University, Medford, MA 02115, United States.
ORCID-Profile: LeAnn M Lindsey, Keith Dufault-Thompson, W Zac Stephens, Xiaofang Jiang
Förderung
- National Science Foundation
- NCATS NIH HHS
- NIH HPC Biowulf cluster
Zitieren
APA
Lindsey, L. M., Pershing, N. L., Habib, A., Dufault-Thompson, K., Stephens, W. Z., Blaschke, A. J., Jiang, X., & Sundar, H. (2025). The impact of tokenizer selection in genomic language models. Bioinformatics. https://doi.org/10.1093/bioinformatics/btaf456
Vancouver
Lindsey LM, Pershing NL, Habib A, Dufault-Thompson K, Stephens WZ, Blaschke AJ, et al. The impact of tokenizer selection in genomic language models. Bioinformatics. 2025. doi:10.1093/bioinformatics/btaf456
BibTeX
@article{Lindsey2025,
author = {Lindsey, LeAnn M and Pershing, Nicole L and Habib, Anisa and Dufault-Thompson, Keith and Stephens, W Zac and Blaschke, Anne J and Jiang, Xiaofang and Sundar, Hari},
title = {The impact of tokenizer selection in genomic language models},
journal = {Bioinformatics},
year = {2025},
doi = {10.1093/bioinformatics/btaf456}
}
EUVIMED