Language-Specific Tokenization for Assamese: Efficiency and Downstream Integration with LLMs
Impact in Computics · 2025
DOI: 10.65500/computics-2025-004 ↗Kurz gefasst
Fachartikel in der Zeitschrift Impact in Computics, 2025. Verfasst von Basab Nath und 1 weiteren Autor:innen. Die Arbeit nennt 22 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Tokenization is a fundamental step in NLP, influencing both computational efficiency and downstream task performance. While subword methods such as Byte-Pair Encoding (BPE), WordPiece, and Unigram have shown strong results for high-resource languages, their suitability for low-resource and morphologically rich languages like Assamese remains insufficiently understood. This study presents a systematic evaluation of these tokenizers on a curated Assamese Wikipedia corpus, examining intrinsic efficiency metrics—including subword fertility, compression ratio, tokenization speed, and token diversity—alongside statistical validation and energy trade-offs. We further connect intrinsic behaviour to practical outcomes by fine-tuning IndicBERT and mBERT on sentiment and hate-speech tasks, and by assessing morphological boundary preservation. Results show that BPE-32K provides the most compact and semantically coherent segmentation, improves downstream F1 scores by 3–4 points, and preserves morpheme boundaries in 82% of cases, while WordPiece-16K offers the fastest tokenization speed. Overall, the findings demonstrate that vocabulary scaling reduces over-segmentation and that language-specific tokenizers substantially outperform multilingual defaults for Assamese. This work provides empirical guidelines for selecting tokenizers tailored to low-resource Indic languages and for integrating them effectively into LLM pipelines.
Zitieren
APA
Nath, B., & Tamang, S. (2025). Language-Specific Tokenization for Assamese: Efficiency and Downstream Integration with LLMs. Impact in Computics. https://doi.org/10.65500/computics-2025-004
Vancouver
Nath B, Tamang S. Language-Specific Tokenization for Assamese: Efficiency and Downstream Integration with LLMs. Impact in Computics. 2025. doi:10.65500/computics-2025-004
BibTeX
@article{Nath2025,
author = {Nath, Basab and Tamang, Sagar},
title = {Language-Specific Tokenization for Assamese: Efficiency and Downstream Integration with LLMs},
journal = {Impact in Computics},
year = {2025},
doi = {10.65500/computics-2025-004}
}
EUVIMED