EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Root, Suffix, or Byte? A Reproducible Study of Sub Word Tokenization for a Morphologically Rich Language (Russian)

Mardiyeva Shoxzoda Totliboy Qizi, Shakhlo Djamaldinova, Sabina Umirzakova

Electronics · 17.09.2026 · 15 (18) · 4236

DOI: 10.3390/electronics15184236 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Electronics, online erschienen am 17.09.2026. Verfasst von Mardiyeva Shoxzoda Totliboy Qizi und 2 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 31 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Tokenization is the first design choice in a language model and one of the most frequently overlooked for morphologically rich languages, as it directly shapes sequence length and how related word forms are segmented into reusable sub word units. Using Russian as a case study, we train sub word tokenizers from scratch on a small, morphology-dense corpus and quantify two effects. First, a coverage penalty: a hypothetical English-centric byte-level tokenizer that has never learned Cyrillic falls back to raw UTF-8 bytes and spends 8.6× more tokens on the same Russian text than a Russian-trained Byte-Pair Encoding (BPE) tokenizer. Second, a compression morphology trade-off as vocabulary grows, BPE fertility falls toward one token per word, but it does so by memorizing complete inflected word forms rather than by isolating morphemes, so a nine-form paradigm of «дoм» (house) is stored as nine unrelated tokens with zero root sharing. A Unigram tokenizer at a comparable budget instead factors words into a shared root plus a shortening, representing the same paradigm with fewer unique token types at the price of slightly higher fertility. We release the corpus, training scripts, and an in-browser reimplementation of the trained BPE model that reproduces the reference tokenizer exactly. The numbers are illustrative, but the mechanisms are real and match findings in the recent Russian NLP literature.

Zitieren

APA

Qizi, M. S. T., Djamaldinova, S., & Umirzakova, S. (2026). Root, Suffix, or Byte? A Reproducible Study of Sub Word Tokenization for a Morphologically Rich Language (Russian). Electronics, 15(18), 4236. https://doi.org/10.3390/electronics15184236

Vancouver

Qizi MST, Djamaldinova S, Umirzakova S. Root, Suffix, or Byte? A Reproducible Study of Sub Word Tokenization for a Morphologically Rich Language (Russian). Electronics. 2026;15(18):4236. doi:10.3390/electronics15184236

BibTeX
@article{Qizi2026,
  author = {Qizi, Mardiyeva Shoxzoda Totliboy and Djamaldinova, Shakhlo and Umirzakova, Sabina},
  title = {Root, Suffix, or Byte? A Reproducible Study of Sub Word Tokenization for a Morphologically Rich Language (Russian)},
  journal = {Electronics},
  year = {2026},
  volume = {15},
  number = {18},
  pages = {4236},
  doi = {10.3390/electronics15184236}
}
Kostenlos starten →