Multi-Head Encoding for Extreme Label Classification
IEEE Transactions on Pattern Analysis and Machine Intelligence · 2025-03 · 47 (3) · 2199-2211
DOI: 10.1109/tpami.2024.3522298 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025-03. Verfasst von Daojun Liang und 3 weiteren Autor:innen. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 64 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Mobility Programme of Sino-German Cooperation and Exchange NSFC.
Abstract (Originalsprache)
The number of categories of instances in the real world is normally huge, and each instance may contain multiple labels. To distinguish these massive labels utilizing machine learning, eXtreme Label Classification (XLC) has been established. However, as the number of categories increases, the number of parameters and nonlinear operations in the classifier also rises. This results in a Classifier Computational Overload Problem (CCOP). To address this, we propose a Multi-Head Encoding (MHE) mechanism, which replaces the vanilla classifier with a multi-head classifier. During the training process, MHE decomposes extreme labels into the product of multiple short local labels, with each head trained on these local labels. During testing, the predicted labels can be directly calculated from the local predictions of each head. This reduces the computational load geometrically. Then, according to the characteristics of different XLC tasks, e.g., single-label, multi-label, and model pretraining tasks, three MHE-based implementations, i.e., Multi-Head Product, Multi-Head Cascade, and Multi-Head Sampling, are proposed to more effectively cope with CCOP. Moreover, we theoretically demonstrate that MHE can achieve performance approximately equivalent to that of the vanilla classifier by generalizing the low-rank approximation problem from Frobenius-norm to Cross-Entropy. Experimental results show that the proposed methods achieve state-of-the-art performance while significantly streamlining the training and inference processes of XLC tasks. The source code has been made public at https://github.com/Anoise/MHE.
Förderung
- National Natural Science Foundation of China
- Mobility Programme of Sino-German Cooperation and Exchange NSFC
Zitieren
APA
Liang, D., Zhang, H., Yuan, D., & Zhang, M. (2025). Multi-Head Encoding for Extreme Label Classification. IEEE Transactions on Pattern Analysis and Machine Intelligence, 47(3), 2199-2211. https://doi.org/10.1109/tpami.2024.3522298
Vancouver
Liang D, Zhang H, Yuan D, Zhang M. Multi-Head Encoding for Extreme Label Classification. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2025;47(3):2199-2211. doi:10.1109/tpami.2024.3522298
BibTeX
@article{Liang2025,
author = {Liang, Daojun and Zhang, Haixia and Yuan, Dongfeng and Zhang, Minggao},
title = {Multi-Head Encoding for Extreme Label Classification},
journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
year = {2025},
volume = {47},
number = {3},
pages = {2199--2211},
doi = {10.1109/tpami.2024.3522298}
}
EUVIMED