EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Honest Teacher Knowledge Distillation: A Framework for Uncertainty-Aware Model Compression Using Gaussian Processes

Arjay R. Alba, Jocelyn F. Villaverde

IEEE Access · 2026 · 14 · 56761-56778

DOI: 10.1109/access.2026.3681278 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Access, 2026. Verfasst von Arjay R. Alba und 1 weiteren Autor:innen. Die Arbeit nennt 47 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Southeast Asian Regional Center for Graduate Study and Research in Agriculture.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

We introduce Honest Teacher Knowledge Distillation (HTKD), an uncertainty-aware distillation framework that transfers both knowledge and confidence from a teacher to a compact student by integrating Gaussian Process (GP)–based predictive uncertainty into the learning objective. Unlike conventional knowledge distillation that treats teacher outputs as uniformly reliable, HTKD explicitly models uncertainty over the teacher’s feature representations and logits, and adaptively down-weights ambiguous supervision. This enables reliability-aware knowledge transfer even in fully supervised settings, where teachers remain imperfect on visually ambiguous or rare samples. HTKD is evaluated using a ResNet- $101~\to $ ResNet-18 teacher–student pair across datasets of varying scale—MS COCO, PASCAL VOC, and Oxford-IIIT Pet—using detection accuracy and calibration metrics. HTKD improves mAP over the non-distilled student by +4.3 on COCO, +4.5 on VOC, and +4.8 on Oxford-IIIT Pet, while achieving the lowest Expected Calibration Error and Brier Score across all datasets. A progressive loss-function ablation confirms monotonic gains from feature-only KD to the full GP-weighted formulation, along with faster and more stable convergence. Robustness is further assessed under Gaussian and motion blur corruptions, where HTKD degrades significantly slower than single-teacher KD and achieves trends comparable to multi-teacher KD, while retaining a single-teacher training cost. These results demonstrate that uncertainty-aware distillation enhances accuracy, calibration, and robustness, making HTKD suitable for reliable edge and real-world deployment.

Förderung

  • Southeast Asian Regional Center for Graduate Study and Research in Agriculture

Zitieren

APA

Alba, A. R., & Villaverde, J. F. (2026). Honest Teacher Knowledge Distillation: A Framework for Uncertainty-Aware Model Compression Using Gaussian Processes. IEEE Access, 14, 56761-56778. https://doi.org/10.1109/access.2026.3681278

Vancouver

Alba AR, Villaverde JF. Honest Teacher Knowledge Distillation: A Framework for Uncertainty-Aware Model Compression Using Gaussian Processes. IEEE Access. 2026;14:56761-56778. doi:10.1109/access.2026.3681278

BibTeX
@article{Alba2026,
  author = {Alba, Arjay R. and Villaverde, Jocelyn F.},
  title = {Honest Teacher Knowledge Distillation: A Framework for Uncertainty-Aware Model Compression Using Gaussian Processes},
  journal = {IEEE Access},
  year = {2026},
  volume = {14},
  pages = {56761--56778},
  doi = {10.1109/access.2026.3681278}
}
Kostenlos starten →