BUILDING A DEEP SEARCH CRAWLER FOR THE KAZAKH LANGUAGE: A REPRODUCIBLE WEB-SCALE PIPELINE
Scientific Journal of Astana IT University · 30.03.2026
DOI: 10.37943/25vbid7988 ↗Kurz gefasst
Fachartikel in der Zeitschrift Scientific Journal of Astana IT University, online erschienen am 30.03.2026. Verfasst von Assel Ospan und 4 weiteren Autor:innen. Die Arbeit nennt 31 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0).
Abstract (Originalsprache)
We present a reproducible, web-scale pipeline for building a Kazakh-language corpus from the national e-government portal. The system treats the website as a directed graph and performs breadth-first traversal to preserve section hierarchy. Static acquisition relies on robust HTTP requests and HTML parsing; for pages with dynamic widgets, we selectively enable a headless layer to render the final DOM prior to extraction. We define a minimal JSON schema aligned with downstream NLP needs (URL, category, titles, cleaned descriptions) and implement normalization (Unicode NFC/NFKC, transliteration repair for Kazakh, boilerplate removal) and fragment-level deduplication. To strengthen the scientific contribution, we formalize the crawling–extraction process as an optimization under resource constraints and propose field-level quality metrics (precision, recall, F1), coverage of categories, and completeness gains attributable to headless rendering. Our experimental protocol compares static parsing against a hybrid static+headless setup on multiple portal categories, reports field-wise effectiveness with confidence intervals, and analyzes dominant error sources (DOM drift, client-side rendering, code-switching). Ablation studies quantify the impact of normalization and duplication. We also outline ethical access (robots.txt compliance, throttling, conditional requests) and provide artifacts to ensure reproducibility (versioned scripts, schema validators, logging). We release open-source scripts, detailed runbooks, and a small, labeled benchmark to facilitate fair comparisons and independent replication across institutions. The resulting corpus targets low-resource Kazakh NLP and e-government analytics, supporting tasks such as classification, terminology normalization, named-entity recognition, and LLM adaptation. Overall, the proposed pipeline demonstrates that selective headless rendering combined with rigorous normalization is a practical and effective strategy for high-quality data acquisition in dynamically rendered public portals.
Zitieren
APA
Ospan, A., Mansurova, M., Nuraliev, F., Khamdamov, R., & Sarsembayeva, T. (2026). BUILDING A DEEP SEARCH CRAWLER FOR THE KAZAKH LANGUAGE: A REPRODUCIBLE WEB-SCALE PIPELINE. Scientific Journal of Astana IT University. https://doi.org/10.37943/25vbid7988
Vancouver
Ospan A, Mansurova M, Nuraliev F, Khamdamov R, Sarsembayeva T. BUILDING A DEEP SEARCH CRAWLER FOR THE KAZAKH LANGUAGE: A REPRODUCIBLE WEB-SCALE PIPELINE. Scientific Journal of Astana IT University. 2026. doi:10.37943/25vbid7988
BibTeX
@article{Ospan2026,
author = {Ospan, Assel and Mansurova, Madina and Nuraliev, Fakhriddin and Khamdamov, Rustam and Sarsembayeva, Talshyn},
title = {BUILDING A DEEP SEARCH CRAWLER FOR THE KAZAKH LANGUAGE: A REPRODUCIBLE WEB-SCALE PIPELINE},
journal = {Scientific Journal of Astana IT University},
year = {2026},
doi = {10.37943/25vbid7988}
}
EUVIMED