EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Efficient Video-to-Audio Generation via Multiple Foundation Models Mapper

Gehui Chen, Guan’an Wang, Xiaowen Huang, Jitao Sang

ACM Transactions on Multimedia Computing, Communications, and Applications · 31.08.2026 · 22 (8) · 1-19

DOI: 10.1145/3820048 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Multimedia Computing, Communications, and Applications, online erschienen am 21.07.2026. Verfasst von Gehui Chen und 3 weiteren Autor:innen. Die Arbeit nennt 53 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Natural Science Foundation of China, National Key Research and Development Program of China, Beijing Natural Science Foundation.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Recent video-to-audio (V2A) generation relies on extracting semantic and temporal features from video to condition generative models. Training these models from scratch is resource-intensive. Consequently, leveraging foundation models (FMs) has gained traction due to their cross-modal knowledge transfer and generalization capabilities. One prior work has explored fine-tuning a lightweight mapper network to connect a pre-trained visual encoder with a text-to-audio generation model for V2A. Inspired by this, we introduce the Multiple Foundation Model Mapper (MFM-Mapper). Compared to the previous mapper approach, MFM-Mapper benefits from richer semantic and temporal information by fusing features from dual visual encoders. Furthermore, by replacing a linear mapper with GPT-2, MFM-Mapper improves feature alignment, drawing parallels between cross-modal features mapping and autoregressive translation tasks. Our MFM-Mapper demonstrates high efficiency in terms of both data requirements and training costs. Compared to prior linear mapper-based work, it achieves superior performance in semantic and temporal consistency with only about 40% of the training data and 40% of the training epochs.

Förderung

  • National Natural Science Foundation of China
  • National Key Research and Development Program of China
  • Beijing Natural Science Foundation
  • Fundamental Research Funds for the Central Universities

Zitieren

APA

Chen, G., Wang, G., Huang, X., & Sang, J. (2026). Efficient Video-to-Audio Generation via Multiple Foundation Models Mapper. ACM Transactions on Multimedia Computing, Communications, and Applications, 22(8), 1-19. https://doi.org/10.1145/3820048

Vancouver

Chen G, Wang G, Huang X, Sang J. Efficient Video-to-Audio Generation via Multiple Foundation Models Mapper. ACM Transactions on Multimedia Computing, Communications, and Applications. 2026;22(8):1-19. doi:10.1145/3820048

BibTeX
@article{Chen2026,
  author = {Chen, Gehui and Wang, Guan’an and Huang, Xiaowen and Sang, Jitao},
  title = {Efficient Video-to-Audio Generation via Multiple Foundation Models Mapper},
  journal = {ACM Transactions on Multimedia Computing, Communications, and Applications},
  year = {2026},
  volume = {22},
  number = {8},
  pages = {1--19},
  doi = {10.1145/3820048}
}
Kostenlos starten →