LaVie: High-Quality Video Generation with Cascaded Latent Diffusion Models
International Journal of Computer Vision · 2025-05 · 133 (5) · 3059-3078
DOI: 10.1007/s11263-024-02295-1 ↗Kurz gefasst
Fachartikel in der Zeitschrift International Journal of Computer Vision, online erschienen am 23.12.2024. Verfasst von Yaohui Wang und 19 weiteren Autor:innen. Die Arbeit wurde bisher 179-mal zitiert (Crossref) und nennt 89 Literaturangaben. Gefördert von: National Key R&D Program China, National Natural Science Foundation of China, Science and Technology Commission of Shanghai Municipality.
Abstract (Originalsprache)
This work aims to learn a high-quality text-to-video (T2V) generative model by leveraging a pre-trained text-to-image (T2I) model as a basis. It is a highly desirable yet challenging task to simultaneously (a) accomplish the synthesis of visually realistic and temporally coherent videos while (b) preserving the strong creative generation nature of the pre-trained T2I model. To this end, we propose LaVie, an integrated video generation framework that operates on cascaded video latent diffusion models, comprising a base T2V model, a temporal interpolation model, and a video super-resolution model. Our key insights are two-fold: (1) We reveal that the incorporation of simple temporal self-attentions, coupled with rotary positional encoding, adequately captures the temporal correlations inherent in video data. (2) Additionally, we validate that the process of joint image-video fine-tuning plays a pivotal role in producing high-quality and creative outcomes. To enhance the performance of LaVie, we contribute a comprehensive and diverse video dataset named Vimeo25M, consisting of 25 million text-video pairs that prioritize quality, diversity, and aesthetic appeal. Extensive experiments demonstrate that LaVie achieves state-of-the-art performance both quantitatively and qualitatively. Furthermore, we showcase the versatility of pre-trained LaVie models in various long video generation and personalized video synthesis applications. Project page: https://github.com/Vchitect/LaVie/.
Förderung
- National Key R&D Program China
- National Natural Science Foundation of China
- Science and Technology Commission of Shanghai Municipality
Zitieren
APA
Wang, Y., Chen, X., Ma, X., Zhou, S., Huang, Z., Wang, Y., Yang, C., He, Y., Yu, J., Yang, P., Guo, Y., Wu, T., Si, C., Jiang, Y., Chen, C., Loy, C. C., Dai, B., Lin, D., Qiao, Y., & Liu, Z. (2025). LaVie: High-Quality Video Generation with Cascaded Latent Diffusion Models. International Journal of Computer Vision, 133(5), 3059-3078. https://doi.org/10.1007/s11263-024-02295-1
Vancouver
Wang Y, Chen X, Ma X, Zhou S, Huang Z, Wang Y, et al. LaVie: High-Quality Video Generation with Cascaded Latent Diffusion Models. International Journal of Computer Vision. 2025;133(5):3059-3078. doi:10.1007/s11263-024-02295-1
BibTeX
@article{Wang2025,
author = {Wang, Yaohui and Chen, Xinyuan and Ma, Xin and Zhou, Shangchen and Huang, Ziqi and Wang, Yi and Yang, Ceyuan and He, Yinan and Yu, Jiashuo and Yang, Peiqing and Guo, Yuwei and Wu, Tianxing and Si, Chenyang and Jiang, Yuming and Chen, Cunjian and Loy, Chen Change and Dai, Bo and Lin, Dahua and Qiao, Yu and Liu, Ziwei},
title = {LaVie: High-Quality Video Generation with Cascaded Latent Diffusion Models},
journal = {International Journal of Computer Vision},
year = {2025},
volume = {133},
number = {5},
pages = {3059--3078},
doi = {10.1007/s11263-024-02295-1}
}
EUVIMED