EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

KFD: Selective Token Filtering and Adaptive Weighting for Efficient Knowledge Distillation

Muzaffer Kaan Yuce, Mehmet Fatih Amasyali

Symmetry · 16.04.2026 · 18 (4) · 667

DOI: 10.3390/sym18040667 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Symmetry, online erschienen am 16.04.2026. Verfasst von Muzaffer Kaan Yuce und 1 weiteren Autor:innen. Die Arbeit nennt 46 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Scientific and Technological Research Council of Turkey.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Knowledge distillation (KD) transfers knowledge from large language models (LLMs) to smaller or similarly sized models in order to obtain efficient yet capable systems. However, performing distillation over all tokens is computationally expensive and may weaken the transfer signal. To address this limitation, Knowledge-Filtered Distillation (KFD) is introduced as a selective distillation approach in which tokens are filtered according to the divergence KL(M2∣M0) between a teacher model (M2) and a base model (M0), while the student model (M1) is also derived from the same base model. Only tokens whose divergence exceeds a predefined threshold are distilled. For the selected tokens, the teacher distribution is normalized over the Top-5 predictions, whereas tokens outside this case receive a label-ranking bonus. The proposed conditional Top-5/bonus target design is shown theoretically to yield a lower label-focused target error than using only Top-5 normalization or only the bonus across all tokens. In addition, the KL and cross-entropy (CE) losses are balanced through a dynamically computed batch-level coefficient α. Experiments on multiple Turkish text datasets show that KFD consistently outperforms CE-only training, achieving higher accuracy with less data and shorter training time. KFD also outperforms entropy-based token selection methods and highlights the role of student initialization in effective knowledge transfer, thereby providing an efficient and scalable distillation framework for teacher–student models of equal size.

Förderung

  • Scientific and Technological Research Council of Turkey

Zitieren

APA

Yuce, M. K., & Amasyali, M. F. (2026). KFD: Selective Token Filtering and Adaptive Weighting for Efficient Knowledge Distillation. Symmetry, 18(4), 667. https://doi.org/10.3390/sym18040667

Vancouver

Yuce MK, Amasyali MF. KFD: Selective Token Filtering and Adaptive Weighting for Efficient Knowledge Distillation. Symmetry. 2026;18(4):667. doi:10.3390/sym18040667

BibTeX
@article{Yuce2026,
  author = {Yuce, Muzaffer Kaan and Amasyali, Mehmet Fatih},
  title = {KFD: Selective Token Filtering and Adaptive Weighting for Efficient Knowledge Distillation},
  journal = {Symmetry},
  year = {2026},
  volume = {18},
  number = {4},
  pages = {667},
  doi = {10.3390/sym18040667}
}
Kostenlos starten →