EUVIMEDEUVIMED entdecken
Crossref

Exploring Audio Cues for Enhanced Test-Time Video Model Adaptation

Runhao Zeng, Qi Deng, Ronghao Zhang, Shuaicheng Niu, Jian Chen, Xiping Hu, Victor C. M. Leung

IEEE Transactions on Circuits and Systems for Video Technology · 2025-12 · 35 (12) · 12292-12305

DOI: 10.1109/tcsvt.2025.3579549 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Circuits and Systems for Video Technology, 2025-12. Verfasst von Runhao Zeng und 6 weiteren Autor:innen. Die Arbeit wurde bisher 7-mal zitiert (Crossref) und nennt 66 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Excellent Science and Technology Creative Talent Training Program of Shenzhen Municipality, Basic and Applied Basic Research Foundation of Guangdong Province.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Test-time adaptation (TTA) aims to boost the generalization capability of a trained model by conducting self-/unsupervised learning during the testing phase. While most existing TTA methods for video primarily utilize visual supervisory signals, they often overlook the potential contribution of inherent audio data. To address this gap, we propose a novel approach that incorporates audio information into video TTA. Our method capitalizes on the rich semantic content of audio to generate audio-assisted pseudo-labels, a new concept in the context of video TTA. Specifically, we propose an audio-to-video label mapping method by first employing pre-trained audio models to classify audio signals extracted from videos and then mapping the audio-based predictions to video label spaces through large language models, thereby establishing a connection between the audio categories and video labels. To effectively leverage the generated pseudo-labels, we present a flexible adaptation cycle that determines the optimal number of adaptation iterations for each sample, based on changes in loss and consistency across different views. This enables a customized adaptation process for each sample. Experimental results on two widely used datasets (UCF101-C and Kinetics-Sounds-C), as well as on two newly constructed audio-video TTA datasets (AVE-C and AVMIT-C) with various corruption types, demonstrate the superiority of our approach. Our method consistently improves adaptation performance across different video classification models and represents a significant step forward in integrating audio information into video TTA. Code: https://github.com/keikeiqi/Audio-Assisted-TTA.

Förderung

  • National Natural Science Foundation of China
  • Excellent Science and Technology Creative Talent Training Program of Shenzhen Municipality
  • Basic and Applied Basic Research Foundation of Guangdong Province
  • Key Scientific Research Project of the Department of Education of Guangdong Province
  • Natural Science Foundation of Guangdong Province

Zitieren

APA

Zeng, R., Deng, Q., Zhang, R., Niu, S., Chen, J., Hu, X., & Leung, V. C. M. (2025). Exploring Audio Cues for Enhanced Test-Time Video Model Adaptation. IEEE Transactions on Circuits and Systems for Video Technology, 35(12), 12292-12305. https://doi.org/10.1109/tcsvt.2025.3579549

Vancouver

Zeng R, Deng Q, Zhang R, Niu S, Chen J, Hu X, et al. Exploring Audio Cues for Enhanced Test-Time Video Model Adaptation. IEEE Transactions on Circuits and Systems for Video Technology. 2025;35(12):12292-12305. doi:10.1109/tcsvt.2025.3579549

BibTeX
@article{Zeng2025,
  author = {Zeng, Runhao and Deng, Qi and Zhang, Ronghao and Niu, Shuaicheng and Chen, Jian and Hu, Xiping and Leung, Victor C. M.},
  title = {Exploring Audio Cues for Enhanced Test-Time Video Model Adaptation},
  journal = {IEEE Transactions on Circuits and Systems for Video Technology},
  year = {2025},
  volume = {35},
  number = {12},
  pages = {12292--12305},
  doi = {10.1109/tcsvt.2025.3579549}
}
Kostenlos starten →