Morphology-Aware Segmentation and Tokenization for Turkic Languages: A CSE-Guided Framework (The Kazakh Case)
Information · 29.01.2026 · 17 (2) · 128
DOI: 10.3390/info17020128 ↗Kurz gefasst
Fachartikel in der Zeitschrift Information, online erschienen am 29.01.2026. Verfasst von Ualsher Tukeyev und 1 weiteren Autor:innen. Die Arbeit wurde bisher 5-mal zitiert (Crossref) und nennt 16 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Ministry of Science and Higher Education of the Republic of Kazakhstan.
Abstract (Originalsprache)
The main challenge of resource-poor languages—namely, the lack of sufficiently large and linguistically informed datasets for training neural models—is addressed in this paper by developing a dataset generation technology based on a Complete Set of Endings (CSE) morphological model for Turkic languages. Building on this technology, we propose a CSE-Guided Framework for morphology-aware statistical tokenization and neural model segmentation, with Kazakh as a case study. Applying the proposed CSE-guided approach to adapt well-known tokenizers for Kazakh leads to measurable reductions in neural model training time (up to approximately 33%) in our experimental setting, primarily due to shorter tokenized sentence lengths. In addition, we extend the SOTA FEMSeg-CRF architecture by incorporating Kazakh vowel–consonant harmony rules at the embedding generation stage. Within the proposed framework, training on a corpus of CSE-generated wordforms results in the FEMSeg_kaz_v2 model, which is evaluated using intrinsic segmentation metrics. Training on a CSE-segmented sentence corpus yields FEMSeg_kaz_v3, which is further assessed using intrinsic, extrinsic, and external evaluation on a manually prepared gold-standard dataset. The paper presents a CSE-guided framework for morphology-aware tokenization and segmentation for Turkic languages, supported by corpus construction, model extensions, and multi-level evaluation. The proposed CSE-Guided Framework can potentially be adapted for other Turkic languages.
Förderung
- Ministry of Science and Higher Education of the Republic of Kazakhstan
Zitieren
APA
Tukeyev, U., & Rysbek, B. (2026). Morphology-Aware Segmentation and Tokenization for Turkic Languages: A CSE-Guided Framework (The Kazakh Case). Information, 17(2), 128. https://doi.org/10.3390/info17020128
Vancouver
Tukeyev U, Rysbek B. Morphology-Aware Segmentation and Tokenization for Turkic Languages: A CSE-Guided Framework (The Kazakh Case). Information. 2026;17(2):128. doi:10.3390/info17020128
BibTeX
@article{Tukeyev2026,
author = {Tukeyev, Ualsher and Rysbek, Bekarys},
title = {Morphology-Aware Segmentation and Tokenization for Turkic Languages: A CSE-Guided Framework (The Kazakh Case)},
journal = {Information},
year = {2026},
volume = {17},
number = {2},
pages = {128},
doi = {10.3390/info17020128}
}
EUVIMED