EUVIMEDEUVIMED entdecken
Crossref

Generalized Kullback–Leibler Divergence Loss

Jiequan Cui, Beier Zhu, Qingshan Xu, Zhuotao Tian, Xiaojuan Qi, Bei Yu, Hanwang Zhang, Richang Hong

IEEE Transactions on Pattern Analysis and Machine Intelligence · 2026-10 · 48 (10) · 13149-13161

DOI: 10.1109/tpami.2026.3703828 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2026-10. Verfasst von Jiequan Cui und 7 weiteren Autor:innen. Die Arbeit wurde bisher 5-mal zitiert (Crossref) und nennt 76 Literaturangaben. Gefördert von: National Natural Science Foundation of China.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

In this paper, we delve deeper into the Kullback-Leibler (KL) Divergence loss and mathematically prove that it is equivalent to the Decoupled Kullback-Leibler (DKL) Divergence loss that consists of 1) a weighted Mean Square Error ($\mathbf {w}$wMSE) loss and 2) a Cross-Entropy loss incorporating soft labels. Thanks to the decoupled structure of DKL loss, we have identified two areas for improvement. Firstly, we address the limitation of KL loss in scenarios like knowledge distillation by breaking its asymmetric optimization property along with a smoother weight function. This modification effectively alleviates convergence challenges in optimization, particularly for classes with high teacher-predicted scores. Secondly, we introduce class-wise global information into KL/DKL to reduce bias arising from individual samples. With these two enhancements, we derive the Generalized Kullback-Leibler (GKL) Divergence loss and evaluate its effectiveness by conducting experiments on CIFAR-10/100, ImageNet, and vision-language datasets, focusing on adversarial training and knowledge distillation tasks. Specifically, we achieve new state-of-the-art adversarial robustness on the public leaderboard - RobustBench and competitive knowledge distillation performance across CIFAR/ImageNet models and CLIP models, demonstrating the substantial practical merits.

Förderung

  • National Natural Science Foundation of China

Zitieren

APA

Cui, J., Zhu, B., Xu, Q., Tian, Z., Qi, X., Yu, B., Zhang, H., & Hong, R. (2026). Generalized Kullback–Leibler Divergence Loss. IEEE Transactions on Pattern Analysis and Machine Intelligence, 48(10), 13149-13161. https://doi.org/10.1109/tpami.2026.3703828

Vancouver

Cui J, Zhu B, Xu Q, Tian Z, Qi X, Yu B, et al. Generalized Kullback–Leibler Divergence Loss. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2026;48(10):13149-13161. doi:10.1109/tpami.2026.3703828

BibTeX
@article{Cui2026,
  author = {Cui, Jiequan and Zhu, Beier and Xu, Qingshan and Tian, Zhuotao and Qi, Xiaojuan and Yu, Bei and Zhang, Hanwang and Hong, Richang},
  title = {Generalized Kullback–Leibler Divergence Loss},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2026},
  volume = {48},
  number = {10},
  pages = {13149--13161},
  doi = {10.1109/tpami.2026.3703828}
}
Kostenlos starten →