EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

The impact of tokenizer selection in genomic language models

LeAnn M Lindsey iD, Nicole L Pershing, Anisa Habib, Keith Dufault-Thompson iD, W Zac Stephens iD, Anne J Blaschke, Xiaofang Jiang iD, Hari Sundar

Bioinformatics · 2025

DOI: 10.1093/bioinformatics/btaf456 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Bioinformatics, 2025. Verfasst von LeAnn M Lindsey und 7 weiteren Autor:innen. Beteiligte Einrichtung: Kahlert School of Computing, University of Utah, Salt Lake City, Utah 84112, United States. Hauptthemen laut MEDLINE: Natural Language Processing, Genomics, Models, Genetic. Die Arbeit wurde bisher 12-mal zitiert (Crossref) und nennt 34 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Science Foundation, NCATS NIH HHS, NIH HPC Biowulf cluster.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Motivation Genomic language models have recently emerged as a new method to decode, interpret, and generate genetic sequences. Existing genomic language models have utilized various tokenization methods, including character tokenization, overlapping and nonoverlapping k-mer tokenization, and byte-pair encoding, a method widely used in natural language models. Genomic sequences differ from natural language because of their low character variability, complex and overlapping features, and inconsistent directionality. These features make subword tokenization in genomic language models significantly different from both traditional language models and protein language models. Results This study explores the impact of tokenization in genomic language models by evaluating their downstream performance on 44 classification fine-tuning tasks. We also perform a direct comparison of byte pair encoding and character tokenization in Mamba, a state-space model. Our results indicate that character tokenization outperforms subword tokenization methods on tasks that rely on nucleotide-level resolution, such as splice site prediction and promoter detection. While byte-pair tokenization had stronger performance on the SARS-CoV-2 variant classification task, we observed limited statistically significant differences between tokenization methods on the remaining downstream tasks. Availability and implementation Detailed results of all benchmarking experiments are available in https://github.com/leannmlindsey/DNAtokenization. Training datasets and pretrained models are available at https://huggingface.co/datasets/leannmlindsey. Datasets and processing scripts are available at doi: 10.5281/zenodo.16287401 and doi: 10.5281/zenodo.16287130.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Natural Language ProcessingGenomicsModels, GeneticSARS-CoV-2HumansCOVID-19Algorithms

Autor:innen und Institutionen

  • Kahlert School of Computing, University of Utah, Salt Lake City, Utah 84112, United States.
  • National Library of Medicine, National Institutes of Health, Bethesda, Maryland 20892, United States.
  • Department of Pediatrics, School of Medicine, University of Utah, Salt Lake City, Utah 84112, United States.
  • Department of Pathology, School of Medicine, University of Utah, Salt Lake City, Utah 84112, United States.
  • Department of Computer Science, Tufts University, Medford, MA 02115, United States.

ORCID-Profile: LeAnn M Lindsey, Keith Dufault-Thompson, W Zac Stephens, Xiaofang Jiang

Förderung

  • National Science Foundation
  • NCATS NIH HHS
  • NIH HPC Biowulf cluster

Zitieren

APA

Lindsey, L. M., Pershing, N. L., Habib, A., Dufault-Thompson, K., Stephens, W. Z., Blaschke, A. J., Jiang, X., & Sundar, H. (2025). The impact of tokenizer selection in genomic language models. Bioinformatics. https://doi.org/10.1093/bioinformatics/btaf456

Vancouver

Lindsey LM, Pershing NL, Habib A, Dufault-Thompson K, Stephens WZ, Blaschke AJ, et al. The impact of tokenizer selection in genomic language models. Bioinformatics. 2025. doi:10.1093/bioinformatics/btaf456

BibTeX
@article{Lindsey2025,
  author = {Lindsey, LeAnn M and Pershing, Nicole L and Habib, Anisa and Dufault-Thompson, Keith and Stephens, W Zac and Blaschke, Anne J and Jiang, Xiaofang and Sundar, Hari},
  title = {The impact of tokenizer selection in genomic language models},
  journal = {Bioinformatics},
  year = {2025},
  doi = {10.1093/bioinformatics/btaf456}
}
Kostenlos starten →