EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC-ND 4.0

SCALABLE NEAR-DUPLICATE DETECTION IN KAZAKH SCIENTIFIC TEXTS VIA SEMANTIC EMBEDDINGS AND OPTIMIZED CANDIDATE FILTERING

Oleksandr Kuchanskyi, Valeriya Kazagasheva, Svitlana Biloshchytska, Dina Kantayeva

Scientific Journal of Astana IT University · 30.03.2026

DOI: 10.37943/25nvvs5297 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Scientific Journal of Astana IT University, online erschienen am 30.03.2026. Verfasst von Oleksandr Kuchanskyi und 3 weiteren Autor:innen. Die Arbeit nennt 22 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

This work considers the problem of efficient detection of near-duplicate documents in Kazakh scientific texts, which is particularly challenging due to the agglutinative nature of the language and the high computational cost of pairwise document comparison. Traditional approaches based on lexical similarity are ineffective under such conditions, while semantic models, although more accurate, are computationally expensive and scale poorly. To overcome these limitations, the study proposes a scalable framework that combines semantic similarity modeling with optimization techniques, including text canonicalization, efficient indexing, and multi-stage candidate filtering. The canonicalization process reduces morphological variability, increasing the stability of similarity estimation for Kazakh texts. The indexing mechanism, based on dense vector representations, enables efficient selection of candidate pairs using approximate nearest neighbor search. The hierarchical filtering strategy further reduces the number of comparisons, while a transformer-based model provides accurate semantic matching. The proposed approach is evaluated on a large-scale dataset of Kazakh scientific abstracts and near-duplicate pairs. The results demonstrate that the framework achieves high detection accuracy while significantly reducing computational costs compared to exhaustive pairwise comparison. The use of dynamic threshold adjustment allows effective handling of overlapping similarity distributions between duplicate and non-duplicate classes. The obtained results confirm that the combination of linguistic preprocessing and computational optimization is crucial for scalable near-duplicate detection in low-resource agglutinative languages such as Kazakh. The proposed framework can be applied in plagiarism detection, document deduplication, and large-scale text analysis systems.

Zitieren

APA

Kuchanskyi, O., Kazagasheva, V., Biloshchytska, S., & Kantayeva, D. (2026). SCALABLE NEAR-DUPLICATE DETECTION IN KAZAKH SCIENTIFIC TEXTS VIA SEMANTIC EMBEDDINGS AND OPTIMIZED CANDIDATE FILTERING. Scientific Journal of Astana IT University. https://doi.org/10.37943/25nvvs5297

Vancouver

Kuchanskyi O, Kazagasheva V, Biloshchytska S, Kantayeva D. SCALABLE NEAR-DUPLICATE DETECTION IN KAZAKH SCIENTIFIC TEXTS VIA SEMANTIC EMBEDDINGS AND OPTIMIZED CANDIDATE FILTERING. Scientific Journal of Astana IT University. 2026. doi:10.37943/25nvvs5297

BibTeX
@article{Kuchanskyi2026,
  author = {Kuchanskyi, Oleksandr and Kazagasheva, Valeriya and Biloshchytska, Svitlana and Kantayeva, Dina},
  title = {SCALABLE NEAR-DUPLICATE DETECTION IN KAZAKH SCIENTIFIC TEXTS VIA SEMANTIC EMBEDDINGS AND OPTIMIZED CANDIDATE FILTERING},
  journal = {Scientific Journal of Astana IT University},
  year = {2026},
  doi = {10.37943/25nvvs5297}
}
Kostenlos starten →