EUVIMEDEUVIMED entdecken
Crossref

Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generation

David Junhao Zhang, Jay Zhangjie Wu, Jia-Wei Liu, Rui Zhao, Lingmin Ran, Yuchao Gu, Difei Gao, Mike Zheng Shou

International Journal of Computer Vision · 2025-04 · 133 (4) · 1879-1893

DOI: 10.1007/s11263-024-02271-9 ↗

Kurz gefasst

Fachartikel in der Zeitschrift International Journal of Computer Vision, online erschienen am 24.10.2024. Verfasst von David Junhao Zhang und 7 weiteren Autor:innen. Die Arbeit wurde bisher 119-mal zitiert (Crossref) und nennt 67 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Significant advancements have been achieved in the realm of large-scale pre-trained text-to-video Diffusion Models (VDMs). However, previous methods either rely solely on pixel-based VDMs, which come with high computational costs, or on latent-based VDMs, which often struggle with precise text-video alignment. In this paper, we are the first to propose a hybrid model, dubbed as Show-1, which marries pixel-based and latent-based VDMs for text-to-video generation. Our model first uses pixel-based VDMs to produce a low-resolution video of strong text-video correlation. After that, we propose a novel expert translation method that employs the latent-based VDMs to further upsample the low-resolution video to high resolution, which can also remove potential artifacts and corruptions from low-resolution videos. Compared to latent VDMs, Show-1 can produce high-quality videos of precise text-video alignment; Compared to pixel VDMs, Show-1 is much more efficient (GPU memory usage during inference is 15 G vs. 72 G). Furthermore, our Show-1 model can be readily adapted for motion customization and video stylization applications through simple temporal attention layer finetuning. Our model achieves state-of-the-art performance on standard video generation benchmarks. Code of Show-1 is publicly available and more videos can be found here.

Zitieren

APA

Zhang, D. J., Wu, J. Z., Liu, J., Zhao, R., Ran, L., Gu, Y., Gao, D., & Shou, M. Z. (2025). Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generation. International Journal of Computer Vision, 133(4), 1879-1893. https://doi.org/10.1007/s11263-024-02271-9

Vancouver

Zhang DJ, Wu JZ, Liu J, Zhao R, Ran L, Gu Y, et al. Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generation. International Journal of Computer Vision. 2025;133(4):1879-1893. doi:10.1007/s11263-024-02271-9

BibTeX
@article{Zhang2025,
  author = {Zhang, David Junhao and Wu, Jay Zhangjie and Liu, Jia-Wei and Zhao, Rui and Ran, Lingmin and Gu, Yuchao and Gao, Difei and Shou, Mike Zheng},
  title = {Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generation},
  journal = {International Journal of Computer Vision},
  year = {2025},
  volume = {133},
  number = {4},
  pages = {1879--1893},
  doi = {10.1007/s11263-024-02271-9}
}
Kostenlos starten →