EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Morphology-Aware Segmentation and Tokenization for Turkic Languages: A CSE-Guided Framework (The Kazakh Case)

Ualsher Tukeyev, Bekarys Rysbek

Information · 29.01.2026 · 17 (2) · 128

DOI: 10.3390/info17020128 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Information, online erschienen am 29.01.2026. Verfasst von Ualsher Tukeyev und 1 weiteren Autor:innen. Die Arbeit wurde bisher 5-mal zitiert (Crossref) und nennt 16 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Ministry of Science and Higher Education of the Republic of Kazakhstan.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The main challenge of resource-poor languages—namely, the lack of sufficiently large and linguistically informed datasets for training neural models—is addressed in this paper by developing a dataset generation technology based on a Complete Set of Endings (CSE) morphological model for Turkic languages. Building on this technology, we propose a CSE-Guided Framework for morphology-aware statistical tokenization and neural model segmentation, with Kazakh as a case study. Applying the proposed CSE-guided approach to adapt well-known tokenizers for Kazakh leads to measurable reductions in neural model training time (up to approximately 33%) in our experimental setting, primarily due to shorter tokenized sentence lengths. In addition, we extend the SOTA FEMSeg-CRF architecture by incorporating Kazakh vowel–consonant harmony rules at the embedding generation stage. Within the proposed framework, training on a corpus of CSE-generated wordforms results in the FEMSeg_kaz_v2 model, which is evaluated using intrinsic segmentation metrics. Training on a CSE-segmented sentence corpus yields FEMSeg_kaz_v3, which is further assessed using intrinsic, extrinsic, and external evaluation on a manually prepared gold-standard dataset. The paper presents a CSE-guided framework for morphology-aware tokenization and segmentation for Turkic languages, supported by corpus construction, model extensions, and multi-level evaluation. The proposed CSE-Guided Framework can potentially be adapted for other Turkic languages.

Förderung

  • Ministry of Science and Higher Education of the Republic of Kazakhstan

Zitieren

APA

Tukeyev, U., & Rysbek, B. (2026). Morphology-Aware Segmentation and Tokenization for Turkic Languages: A CSE-Guided Framework (The Kazakh Case). Information, 17(2), 128. https://doi.org/10.3390/info17020128

Vancouver

Tukeyev U, Rysbek B. Morphology-Aware Segmentation and Tokenization for Turkic Languages: A CSE-Guided Framework (The Kazakh Case). Information. 2026;17(2):128. doi:10.3390/info17020128

BibTeX
@article{Tukeyev2026,
  author = {Tukeyev, Ualsher and Rysbek, Bekarys},
  title = {Morphology-Aware Segmentation and Tokenization for Turkic Languages: A CSE-Guided Framework (The Kazakh Case)},
  journal = {Information},
  year = {2026},
  volume = {17},
  number = {2},
  pages = {128},
  doi = {10.3390/info17020128}
}
Kostenlos starten →