EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Language-Specific Tokenization for Assamese: Efficiency and Downstream Integration with LLMs

Basab Nath, Sagar Tamang

Impact in Computics · 2025

DOI: 10.65500/computics-2025-004 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Impact in Computics, 2025. Verfasst von Basab Nath und 1 weiteren Autor:innen. Die Arbeit nennt 22 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Tokenization is a fundamental step in NLP, influencing both computational efficiency and downstream task performance. While subword methods such as Byte-Pair Encoding (BPE), WordPiece, and Unigram have shown strong results for high-resource languages, their suitability for low-resource and morphologically rich languages like Assamese remains insufficiently understood. This study presents a systematic evaluation of these tokenizers on a curated Assamese Wikipedia corpus, examining intrinsic efficiency metrics—including subword fertility, compression ratio, tokenization speed, and token diversity—alongside statistical validation and energy trade-offs. We further connect intrinsic behaviour to practical outcomes by fine-tuning IndicBERT and mBERT on sentiment and hate-speech tasks, and by assessing morphological boundary preservation. Results show that BPE-32K provides the most compact and semantically coherent segmentation, improves downstream F1 scores by 3–4 points, and preserves morpheme boundaries in 82% of cases, while WordPiece-16K offers the fastest tokenization speed. Overall, the findings demonstrate that vocabulary scaling reduces over-segmentation and that language-specific tokenizers substantially outperform multilingual defaults for Assamese. This work provides empirical guidelines for selecting tokenizers tailored to low-resource Indic languages and for integrating them effectively into LLM pipelines.

Zitieren

APA

Nath, B., & Tamang, S. (2025). Language-Specific Tokenization for Assamese: Efficiency and Downstream Integration with LLMs. Impact in Computics. https://doi.org/10.65500/computics-2025-004

Vancouver

Nath B, Tamang S. Language-Specific Tokenization for Assamese: Efficiency and Downstream Integration with LLMs. Impact in Computics. 2025. doi:10.65500/computics-2025-004

BibTeX
@article{Nath2025,
  author = {Nath, Basab and Tamang, Sagar},
  title = {Language-Specific Tokenization for Assamese: Efficiency and Downstream Integration with LLMs},
  journal = {Impact in Computics},
  year = {2025},
  doi = {10.65500/computics-2025-004}
}
Kostenlos starten →