EUVIMEDEUVIMED entdecken
Crossref

Knowledge Distillation for Embeddings of Low-Resource Turkic Family Languages

Mirakram Aghalarov, Elvin Mammadov, Kavsar Huseynova

Gazi University Journal of Science Part A: Engineering and Innovation · 31.03.2026 · 13 (1) · 334-347

DOI: 10.54287/gujsa.1844025 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Gazi University Journal of Science Part A: Engineering and Innovation, online erschienen am 31.03.2026. Verfasst von Mirakram Aghalarov und 2 weiteren Autor:innen. Die Arbeit nennt 20 Literaturangaben. Gefördert von: Baku Higher oil School.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Embedding models are a critical component of Retrieval-Augmented Generation (RAG) systems, but creating efficient, high-performance models for low-resource languages like Azerbaijani presents a significant challenge. To address this, we propose a multi-task distillation framework that improves a target low-resource language by distilling knowledge from English, while crucially preserving the model's original English language performance. Our approach combines cross-lingual knowledge distillation with a bilingual training objective, controlled by a single hyperparameter that balances performance across languages. We release a large-scale English-Azerbaijani parallel corpus of 300,000 Wikipedia articles to facilitate further research. Our experimental results demonstrate that models trained with our framework achieve state-of-the-art performance on Azerbaijani benchmarks: our BGE M3 model improves Azerbaijani STS from 79.3 to 85.0 (+5.7 points) while retaining 98.6% of its original English MTEB performance. In a real-world RAG application for Azerbaijani legal document retrieval, our best model achieves 89% top-1 retrieval accuracy and a 94% "Excellent" rating from domain experts, significantly outperforming baseline models. This framework provides a generalizable blueprint for enhancing embedding models for other low-resource languages, particularly within the Turkic language family.

Förderung

  • Baku Higher oil School

Zitieren

APA

Aghalarov, M., Mammadov, E., & Huseynova, K. (2026). Knowledge Distillation for Embeddings of Low-Resource Turkic Family Languages. Gazi University Journal of Science Part A: Engineering and Innovation, 13(1), 334-347. https://doi.org/10.54287/gujsa.1844025

Vancouver

Aghalarov M, Mammadov E, Huseynova K. Knowledge Distillation for Embeddings of Low-Resource Turkic Family Languages. Gazi University Journal of Science Part A: Engineering and Innovation. 2026;13(1):334-347. doi:10.54287/gujsa.1844025

BibTeX
@article{Aghalarov2026,
  author = {Aghalarov, Mirakram and Mammadov, Elvin and Huseynova, Kavsar},
  title = {Knowledge Distillation for Embeddings of Low-Resource Turkic Family Languages},
  journal = {Gazi University Journal of Science Part A: Engineering and Innovation},
  year = {2026},
  volume = {13},
  number = {1},
  pages = {334--347},
  doi = {10.54287/gujsa.1844025}
}
Kostenlos starten →