Grouped self-attention and quantum-enhanced knowledge distillation
Physica Scripta · 19.06.2026 · 101 (24) · 246012
DOI: 10.1088/1402-4896/ae7657 ↗Kurz gefasst
Fachartikel in der Zeitschrift Physica Scripta, online erschienen am 17.06.2026. Verfasst von Dongfen Li und 5 weiteren Autor:innen. Die Arbeit nennt 52 Literaturangaben. Gefördert von: Sichuan Provincial Science and Technology Plan "Unveiling and Leading" Project, Open Project Fund of the Network and Data Security Key Laboratory of Sichuan Province, Sichuan Provincial Department of Economy and Information Technology Industrial Infrastructure Research Project.
Abstract (Originalsprache)
While knowledge distillation (KD) has proven effective for model compression, existing methods often struggle to capture multi-scale semantic structures and suffer from feature redundancy. To overcome these limitations, we propose grouped self-attention (GSA) and quantum-perception-KD, a multi-scale distillation framework that synergizes GSA with quantum feature enhancement. First, we introduce a GSA mechanism into the intermediate layers, enabling the student model to explicitly perceive spatial dependencies across different scales. Simultaneously, a quantum-perception feature enhancement module is designed as an auxiliary branch, leveraging quantum superposition mechanisms to align complex high-level semantics that are typically challenging for lightweight models. Furthermore, to ensure feature compactness, we incorporate a feature covariance decoupling mechanism that enforces statistical channel independence, thereby reducing information redundancy. Experimental results demonstrate that the proposed method outperforms existing distillation techniques on multiple image classification tasks, validating its effectiveness and application potential.
Förderung
- Sichuan Provincial Science and Technology Plan "Unveiling and Leading" Project
- Open Project Fund of the Network and Data Security Key Laboratory of Sichuan Province
- Sichuan Provincial Department of Economy and Information Technology Industrial Infrastructure Research Project
- Artificial Intelligence Research Fund of Chengdu University of Technology
- Chengdu Bamo Technology Co., Ltd. Commissioned Scientific Research Project
- Sichuan Science and Technology Program
- National Natural Science Foundation of China
- Sichuan Lingke Automobile Manufacturing Co., Ltd.
Zitieren
APA
Li, D., Song, Z., Xiao, P., Zhou, Q., Zhao, P., & Liu, N. (2026). Grouped self-attention and quantum-enhanced knowledge distillation. Physica Scripta, 101(24), 246012. https://doi.org/10.1088/1402-4896/ae7657
Vancouver
Li D, Song Z, Xiao P, Zhou Q, Zhao P, Liu N. Grouped self-attention and quantum-enhanced knowledge distillation. Physica Scripta. 2026;101(24):246012. doi:10.1088/1402-4896/ae7657
BibTeX
@article{Li2026,
author = {Li, Dongfen and Song, Zhengyu and Xiao, Peng and Zhou, Qing and Zhao, Pengbiao and Liu, Nan},
title = {Grouped self-attention and quantum-enhanced knowledge distillation},
journal = {Physica Scripta},
year = {2026},
volume = {101},
number = {24},
pages = {246012},
doi = {10.1088/1402-4896/ae7657}
}
EUVIMED