Efficient Video-to-Audio Generation via Multiple Foundation Models Mapper
ACM Transactions on Multimedia Computing, Communications, and Applications · 31.08.2026 · 22 (8) · 1-19
DOI: 10.1145/3820048 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Multimedia Computing, Communications, and Applications, online erschienen am 21.07.2026. Verfasst von Gehui Chen und 3 weiteren Autor:innen. Die Arbeit nennt 53 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Natural Science Foundation of China, National Key Research and Development Program of China, Beijing Natural Science Foundation.
Abstract (Originalsprache)
Recent video-to-audio (V2A) generation relies on extracting semantic and temporal features from video to condition generative models. Training these models from scratch is resource-intensive. Consequently, leveraging foundation models (FMs) has gained traction due to their cross-modal knowledge transfer and generalization capabilities. One prior work has explored fine-tuning a lightweight mapper network to connect a pre-trained visual encoder with a text-to-audio generation model for V2A. Inspired by this, we introduce the Multiple Foundation Model Mapper (MFM-Mapper). Compared to the previous mapper approach, MFM-Mapper benefits from richer semantic and temporal information by fusing features from dual visual encoders. Furthermore, by replacing a linear mapper with GPT-2, MFM-Mapper improves feature alignment, drawing parallels between cross-modal features mapping and autoregressive translation tasks. Our MFM-Mapper demonstrates high efficiency in terms of both data requirements and training costs. Compared to prior linear mapper-based work, it achieves superior performance in semantic and temporal consistency with only about 40% of the training data and 40% of the training epochs.
Förderung
- National Natural Science Foundation of China
- National Key Research and Development Program of China
- Beijing Natural Science Foundation
- Fundamental Research Funds for the Central Universities
Zitieren
APA
Chen, G., Wang, G., Huang, X., & Sang, J. (2026). Efficient Video-to-Audio Generation via Multiple Foundation Models Mapper. ACM Transactions on Multimedia Computing, Communications, and Applications, 22(8), 1-19. https://doi.org/10.1145/3820048
Vancouver
Chen G, Wang G, Huang X, Sang J. Efficient Video-to-Audio Generation via Multiple Foundation Models Mapper. ACM Transactions on Multimedia Computing, Communications, and Applications. 2026;22(8):1-19. doi:10.1145/3820048
BibTeX
@article{Chen2026,
author = {Chen, Gehui and Wang, Guan’an and Huang, Xiaowen and Sang, Jitao},
title = {Efficient Video-to-Audio Generation via Multiple Foundation Models Mapper},
journal = {ACM Transactions on Multimedia Computing, Communications, and Applications},
year = {2026},
volume = {22},
number = {8},
pages = {1--19},
doi = {10.1145/3820048}
}
EUVIMED