Impact of Tokenization on Language Models: An Analysis for Turkish
ACM Transactions on Asian and Low-Resource Language Information Processing · 30.04.2023 · 22 (4) · 1-21
DOI: 10.1145/3578707 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Asian and Low-Resource Language Information Processing, online erschienen am 25.03.2023. Verfasst von Cagri Toraman und 3 weiteren Autor:innen. Die Arbeit wurde bisher 65-mal zitiert (Crossref) und nennt 67 Literaturangaben.
Abstract (Originalsprache)
Tokenization is an important text preprocessing step to prepare input tokens for deep language models. WordPiece and BPE are de facto methods employed by important models, such as BERT and GPT. However, the impact of tokenization can be different for morphologically rich languages, such as Turkic languages, in which many words can be generated by adding prefixes and suffixes. We compare five tokenizers at different granularity levels, that is, their outputs vary from the smallest pieces of characters to the surface form of words, including a Morphological-level tokenizer. We train these tokenizers and pretrain medium-sized language models using the RoBERTa pretraining procedure on the Turkish split of the OSCAR corpus. We then fine-tune our models on six downstream tasks. Our experiments, supported by statistical tests, reveal that the morphological-level tokenizer delivers a challenging performance with de facto tokenizers. Furthermore, we find that increasing the vocabulary size improves the performance of Morphological- and Word-level tokenizers more than that of de facto tokenizers. The ratio of the number of vocabulary parameters to the total number of model parameters can be empirically chosen as 20% for de facto tokenizers and 40% for other tokenizers to obtain a reasonable trade-off between model size and performance.
Zitieren
APA
Toraman, C., Yilmaz, E. H., Şahi̇nuç, F., & Ozcelik, O. (2023). Impact of Tokenization on Language Models: An Analysis for Turkish. ACM Transactions on Asian and Low-Resource Language Information Processing, 22(4), 1-21. https://doi.org/10.1145/3578707
Vancouver
Toraman C, Yilmaz EH, Şahi̇nuç F, Ozcelik O. Impact of Tokenization on Language Models: An Analysis for Turkish. ACM Transactions on Asian and Low-Resource Language Information Processing. 2023;22(4):1-21. doi:10.1145/3578707
BibTeX
@article{Toraman2023,
author = {Toraman, Cagri and Yilmaz, Eyup Halit and Şahi̇nuç, Furkan and Ozcelik, Oguzhan},
title = {Impact of Tokenization on Language Models: An Analysis for Turkish},
journal = {ACM Transactions on Asian and Low-Resource Language Information Processing},
year = {2023},
volume = {22},
number = {4},
pages = {1--21},
doi = {10.1145/3578707}
}
EUVIMED