EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

SupraTok: Cross-Boundary Tokenization for Enhanced Language Model Performance

Andrei-Valentin Tănase, Elena Pelican

Transactions of the Association for Computational Linguistics · 17.07.2026 · 14 · 1787-1802

DOI: 10.1162/tacl.a.774 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Transactions of the Association for Computational Linguistics, online erschienen am 17.07.2026. Verfasst von Andrei-Valentin Tănase und 1 weiteren Autor:innen. Die Arbeit nennt 24 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Tokenization remains a persistent bottleneck in language modeling, especially when vocabulary learning is limited by whitespace boundaries. We present SupraTok, a tokenizer that crosses whitespace boundaries using three modular components: optional entropy-based data curation, staged curriculum training with PMI-guided candidate search, and multilingual script handling. At 100k vocabulary on the same unfiltered training data, SupraTok improves compression over standard BPE by 17.5% and over the official SuperBPE implementation by 1.8%, while training 2.1× faster than SuperBPE. Across 50k–300k vocabularies in the same matched setting, Supra-Tok remains ahead of SuperBPE by 1.8%–8.6%. We evaluate entropy filtering separately as a pipeline step: at 100k vocabulary it raises SupraTok from 5.78 to 5.99 C/T, while matched controls show a smaller gain for SuperBPE and almost no change for SP-BPE-CrossBoundary. On FLORES-200 across 14 languages, SupraTok yields a macro-averaged 34.9% relative gain over the BPE baseline. In separate downstream experiments with matched compute and fixed token budgets, using 12L-768d and 24L-1024d GPT-2-style backbones with 256k vocabularies, SupraTok improves HellaSwag and MMLU. Overall, these results show that crossing whitespace boundaries gives consistent compression gains under controlled public-data comparisons, while optional entropy filtering provides a separate pipeline benefit.

Zitieren

APA

Tănase, A., & Pelican, E. (2026). SupraTok: Cross-Boundary Tokenization for Enhanced Language Model Performance. Transactions of the Association for Computational Linguistics, 14, 1787-1802. https://doi.org/10.1162/tacl.a.774

Vancouver

Tănase A, Pelican E. SupraTok: Cross-Boundary Tokenization for Enhanced Language Model Performance. Transactions of the Association for Computational Linguistics. 2026;14:1787-1802. doi:10.1162/tacl.a.774

BibTeX
@article{Tnase2026,
  author = {Tănase, Andrei-Valentin and Pelican, Elena},
  title = {SupraTok: Cross-Boundary Tokenization for Enhanced Language Model Performance},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2026},
  volume = {14},
  pages = {1787--1802},
  doi = {10.1162/tacl.a.774}
}
Kostenlos starten →