EUVIMEDEUVIMED entdecken
Crossref

FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Sounds

Yiming Zhang, Yicheng Gu, Yanhong Zeng, Zhening Xing, Yuancheng Wang, Zhizheng Wu, Bin Liu, Kai Chen

International Journal of Computer Vision · 2026-01 · 134 (1)

DOI: 10.1007/s11263-025-02649-3 ↗

Kurz gefasst

Fachartikel in der Zeitschrift International Journal of Computer Vision, online erschienen am 07.01.2026. Verfasst von Yiming Zhang und 7 weiteren Autor:innen. Die Arbeit wurde bisher 24-mal zitiert (Crossref) und nennt 49 Literaturangaben. Gefördert von: National Key R&D Program of China, NSFC, Shenzhen Science and Technology Program.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

We study Neural Foley, the automatic generation of high-quality sound effects synchronizing with videos, enabling an immersive audio-visual experience. Despite its wide range of applications, existing approaches encounter limitations when it comes to simultaneously synthesizing high-quality and video-aligned (i.e.,semantic relevant and temporal synchronized) sounds. To overcome these limitations, we propose FoleyCrafter, a novel framework that leverages a pre-trained text-to-audio model to ensure high-quality audio generation. FoleyCrafter comprises two key components: a semantic adapter for semantic alignment and a temporal adapter for precise audio-video synchronization. The semantic adapter utilizes parallel cross-attention layers to condition audio generation on video features, producing realistic sound effects that are semantically relevant to the visual content. Meanwhile, the temporal adapter estimates time-varying signals from the videos and subsequently synchronizes audio generation with those estimates, leading to enhanced temporal alignment between audio and video. To further enhance the training of the temporal adapter, we introduce a high-quality audio-video synchronized dataset with strong, clear visual cues closely tied to their corresponding audio sounds, TikTokSound. We conduct extensive quantitative and qualitative experiments on standard benchmarks to verify the effectiveness of FoleyCrafter. Models and codes will be available.

Förderung

  • National Key R&D Program of China
  • NSFC
  • Shenzhen Science and Technology Program

Zitieren

APA

Zhang, Y., Gu, Y., Zeng, Y., Xing, Z., Wang, Y., Wu, Z., Liu, B., & Chen, K. (2026). FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Sounds. International Journal of Computer Vision, 134(1). https://doi.org/10.1007/s11263-025-02649-3

Vancouver

Zhang Y, Gu Y, Zeng Y, Xing Z, Wang Y, Wu Z, et al. FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Sounds. International Journal of Computer Vision. 2026;134(1). doi:10.1007/s11263-025-02649-3

BibTeX
@article{Zhang2026,
  author = {Zhang, Yiming and Gu, Yicheng and Zeng, Yanhong and Xing, Zhening and Wang, Yuancheng and Wu, Zhizheng and Liu, Bin and Chen, Kai},
  title = {FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Sounds},
  journal = {International Journal of Computer Vision},
  year = {2026},
  volume = {134},
  number = {1},
  doi = {10.1007/s11263-025-02649-3}
}
Kostenlos starten →