Knowledge Distillation for Embeddings of Low-Resource Turkic Family Languages
Gazi University Journal of Science Part A: Engineering and Innovation · 31.03.2026 · 13 (1) · 334-347
DOI: 10.54287/gujsa.1844025 ↗Kurz gefasst
Fachartikel in der Zeitschrift Gazi University Journal of Science Part A: Engineering and Innovation, online erschienen am 31.03.2026. Verfasst von Mirakram Aghalarov und 2 weiteren Autor:innen. Die Arbeit nennt 20 Literaturangaben. Gefördert von: Baku Higher oil School.
Abstract (Originalsprache)
Embedding models are a critical component of Retrieval-Augmented Generation (RAG) systems, but creating efficient, high-performance models for low-resource languages like Azerbaijani presents a significant challenge. To address this, we propose a multi-task distillation framework that improves a target low-resource language by distilling knowledge from English, while crucially preserving the model's original English language performance. Our approach combines cross-lingual knowledge distillation with a bilingual training objective, controlled by a single hyperparameter that balances performance across languages. We release a large-scale English-Azerbaijani parallel corpus of 300,000 Wikipedia articles to facilitate further research. Our experimental results demonstrate that models trained with our framework achieve state-of-the-art performance on Azerbaijani benchmarks: our BGE M3 model improves Azerbaijani STS from 79.3 to 85.0 (+5.7 points) while retaining 98.6% of its original English MTEB performance. In a real-world RAG application for Azerbaijani legal document retrieval, our best model achieves 89% top-1 retrieval accuracy and a 94% "Excellent" rating from domain experts, significantly outperforming baseline models. This framework provides a generalizable blueprint for enhancing embedding models for other low-resource languages, particularly within the Turkic language family.
Förderung
- Baku Higher oil School
Zitieren
APA
Aghalarov, M., Mammadov, E., & Huseynova, K. (2026). Knowledge Distillation for Embeddings of Low-Resource Turkic Family Languages. Gazi University Journal of Science Part A: Engineering and Innovation, 13(1), 334-347. https://doi.org/10.54287/gujsa.1844025
Vancouver
Aghalarov M, Mammadov E, Huseynova K. Knowledge Distillation for Embeddings of Low-Resource Turkic Family Languages. Gazi University Journal of Science Part A: Engineering and Innovation. 2026;13(1):334-347. doi:10.54287/gujsa.1844025
BibTeX
@article{Aghalarov2026,
author = {Aghalarov, Mirakram and Mammadov, Elvin and Huseynova, Kavsar},
title = {Knowledge Distillation for Embeddings of Low-Resource Turkic Family Languages},
journal = {Gazi University Journal of Science Part A: Engineering and Innovation},
year = {2026},
volume = {13},
number = {1},
pages = {334--347},
doi = {10.54287/gujsa.1844025}
}
EUVIMED