EUVIMEDEUVIMED entdecken
Crossref

V2Meow: Meowing to the Visual Beat via Video-to-Music Generation

Kun Su, Judith Yue Li, Qingqing Huang, Dima Kuzmin, Joonseok Lee, Chris Donahue, Fei Sha, Aren Jansen, Yu Wang, Mauro Verzetti, Timo Denk

Proceedings of the AAAI Conference on Artificial Intelligence · 24.03.2024 · 38 (5) · 4952-4960

DOI: 10.1609/aaai.v38i5.28299 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 24.03.2024. Verfasst von Kun Su und 10 weiteren Autor:innen. Die Arbeit wurde bisher 23-mal zitiert (Crossref).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Video-to-music generation demands both a temporally localized high-quality listening experience and globally aligned video-acoustic signatures. While recent music generation models excel at the former through advanced audio codecs, the exploration of video-acoustic signatures has been confined to specific visual scenarios. In contrast, our research confronts the challenge of learning globally aligned signatures between video and music directly from paired music and videos, without explicitly modeling domain-specific rhythmic or semantic relationships. We propose V2Meow, a video-to-music generation system capable of producing high-quality music audio for a diverse range of video input types using a multi-stage autoregressive model. Trained on 5k hours of music audio clips paired with video frames mined from in-the-wild music videos, V2Meow is competitive with previous domain-specific models when evaluated in a zero-shot manner. It synthesizes high-fidelity music audio waveforms solely by conditioning on pre-trained general-purpose visual features extracted from video frames, with optional style control via text prompts. Through both qualitative and quantitative evaluations, we demonstrate that our model outperforms various existing music generation systems in terms of visual-audio correspondence and audio quality. Music samples are available at tinyurl.com/v2meow.

Zitieren

APA

Su, K., Li, J. Y., Huang, Q., Kuzmin, D., Lee, J., Donahue, C., Sha, F., Jansen, A., Wang, Y., Verzetti, M., & Denk, T. (2024). V2Meow: Meowing to the Visual Beat via Video-to-Music Generation. Proceedings of the AAAI Conference on Artificial Intelligence, 38(5), 4952-4960. https://doi.org/10.1609/aaai.v38i5.28299

Vancouver

Su K, Li JY, Huang Q, Kuzmin D, Lee J, Donahue C, et al. V2Meow: Meowing to the Visual Beat via Video-to-Music Generation. Proceedings of the AAAI Conference on Artificial Intelligence. 2024;38(5):4952-4960. doi:10.1609/aaai.v38i5.28299

BibTeX
@article{Su2024,
  author = {Su, Kun and Li, Judith Yue and Huang, Qingqing and Kuzmin, Dima and Lee, Joonseok and Donahue, Chris and Sha, Fei and Jansen, Aren and Wang, Yu and Verzetti, Mauro and Denk, Timo},
  title = {V2Meow: Meowing to the Visual Beat via Video-to-Music Generation},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2024},
  volume = {38},
  number = {5},
  pages = {4952--4960},
  doi = {10.1609/aaai.v38i5.28299}
}
Kostenlos starten →