EUVIMEDEUVIMED entdecken
Crossref

Mettle: Meta-Token Learning for Memory-Efficient Audio-Visual Adaptation

Jinxing Zhou, Zhihui Li, Yongqiang Yu, Yanghao Zhou, Ruohao Guo, Guangyao Li, Yuxin Mao, Mingfei Han, Xiaojun Chang, Meng Wang

IEEE Transactions on Pattern Analysis and Machine Intelligence · 2026-04 · 48 (4) · 4222-4238

DOI: 10.1109/tpami.2025.3642821 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2026-04. Verfasst von Jinxing Zhou und 9 weiteren Autor:innen. Die Arbeit wurde bisher 8-mal zitiert (Crossref) und nennt 102 Literaturangaben. Gefördert von: National Natural Science Foundation of China, New Generation Artificial Intelligence-National Science and Technology Major Project, Outstanding Youth Fund of Shandong Province.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Mainstream research in audio-visual learning has focused on designing task-specific expert models, primarily implemented through sophisticated multimodal fusion approaches. Recently, a few efforts have aimed to develop more task-independent or universal audiovisual embedding networks, encoding advanced representations for use in various audiovisual downstream tasks. This is typically achieved by fine-tuning large pretrained transformers, such as Swin-V2-L and HTS-AT, in a parameter-efficient manner through techniques such as tuning only a few adapter layers inserted into the pretrained transformer backbone. Although these methods are parameter-efficient, they suffer from significant training memory consumption due to gradient backpropagation through the deep transformer backbones, which limits accessibility for researchers with constrained computational resources. In this paper, we present Meta-Token Learning (Mettle), a simple and memory-efficient method for adapting large-scale pretrained transformer models to downstream audio-visual tasks. Instead of sequentially modifying the output feature distribution of the transformer backbone, Mettle utilizes a lightweight Layer-Centric Distillation (LCD) module to distill in parallel the intact audio or visual features embedded by each transformer layer into compact meta-tokens. This distillation process considers both pretrained knowledge preservation and task-specific adaptation. The obtained meta-tokens can be directly applied to classification tasks, such as audio-visual event localization and audio-visual video parsing. To further support fine-grained segmentation tasks, such as audio-visual segmentation, we introduce a Meta-Token Injection (MTI) module, which utilizes the audio and visual meta-tokens distilled from the top transformer layer to guide feature adaptation in earlier layers. Extensive experiments on multiple audiovisual benchmarks demonstrate that our method significantly reduces memory usage and training time while maintaining parameter efficiency and competitive accuracy.

Förderung

  • National Natural Science Foundation of China
  • New Generation Artificial Intelligence-National Science and Technology Major Project
  • Outstanding Youth Fund of Shandong Province

Zitieren

APA

Zhou, J., Li, Z., Yu, Y., Zhou, Y., Guo, R., Li, G., Mao, Y., Han, M., Chang, X., & Wang, M. (2026). Mettle: Meta-Token Learning for Memory-Efficient Audio-Visual Adaptation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 48(4), 4222-4238. https://doi.org/10.1109/tpami.2025.3642821

Vancouver

Zhou J, Li Z, Yu Y, Zhou Y, Guo R, Li G, et al. Mettle: Meta-Token Learning for Memory-Efficient Audio-Visual Adaptation. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2026;48(4):4222-4238. doi:10.1109/tpami.2025.3642821

BibTeX
@article{Zhou2026,
  author = {Zhou, Jinxing and Li, Zhihui and Yu, Yongqiang and Zhou, Yanghao and Guo, Ruohao and Li, Guangyao and Mao, Yuxin and Han, Mingfei and Chang, Xiaojun and Wang, Meng},
  title = {Mettle: Meta-Token Learning for Memory-Efficient Audio-Visual Adaptation},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2026},
  volume = {48},
  number = {4},
  pages = {4222--4238},
  doi = {10.1109/tpami.2025.3642821}
}
Kostenlos starten →