EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

UniVidX: A Unified Multimodal Framework for Versatile Video Generation via Diffusion Priors

Houyuan Chen, Hong Li, Xianghao Kong, Tianrui Zhu, Shaocong Xu, Weiqing Xiao, Yuwei Guo, Chongjie Ye, Lvmin Zhang, Hao Zhao, Anyi Rao

ACM Transactions on Graphics · 03.07.2026 · 45 (4) · 1-17

DOI: 10.1145/3811304 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Graphics, online erschienen am 03.07.2026. Verfasst von Houyuan Chen und 10 weiteren Autor:innen. Die Arbeit nennt 121 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: NSFC/RGC Collaborative Research Scheme.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Recent progress has shown that video diffusion models (VDMs) can be repurposed to solve various multimodal graphics tasks. However, existing approaches predominantly train separate models for each specific problem setting. This practice locks models into fixed input-output mappings, and typically ignores the joint correlations across modalities. In this paper, we present UniVidX , a unified multimodal framework designed to leverage VDM priors to enable versatile video generation. Our goal is to (i) master diverse pixel-aligned tasks by formulating them as conditional generation problems within multimodal space, (ii) adapt to modality-specific distributions without compromising the backbone's native priors, and (iii) ensure cross-modal consistency during synthesis. Concretely, we propose three key designs: 1) Stochastic Condition Masking (SCM): by randomly partitioning modalities into clean conditions and noisy targets during training, we enable the model to learn omni-directional conditional generation rather than fixed mappings. 2) Decoupled Gated LoRA (DGL): we attach per-modality LoRAs and activate them when a modality serves as a generation target, thereby preserving the VDM's strong priors. 3) Cross-Modal Self-Attention (CMSA): we explicitly share keys/values across modalities while maintaining modality-specific queries, facilitating information exchange and inter-modal alignment. We validate our framework by instantiating it in two domains: 1) UniVid-Intrinsic for RGB videos and their intrinsic maps (albedo, irradiance, normal), and 2) UniVid-Alpha for blended RGB videos and their constituent RGBA layers. Experimental results demonstrate that both models achieve performance competitive with state-of-the-art methods across distinct tasks. Notably, they exhibit robust generalization capabilities in in-the-wild scenarios, even when trained on limited datasets of fewer than 1k videos.

Förderung

  • NSFC/RGC Collaborative Research Scheme

Zitieren

APA

Chen, H., Li, H., Kong, X., Zhu, T., Xu, S., Xiao, W., Guo, Y., Ye, C., Zhang, L., Zhao, H., & Rao, A. (2026). UniVidX: A Unified Multimodal Framework for Versatile Video Generation via Diffusion Priors. ACM Transactions on Graphics, 45(4), 1-17. https://doi.org/10.1145/3811304

Vancouver

Chen H, Li H, Kong X, Zhu T, Xu S, Xiao W, et al. UniVidX: A Unified Multimodal Framework for Versatile Video Generation via Diffusion Priors. ACM Transactions on Graphics. 2026;45(4):1-17. doi:10.1145/3811304

BibTeX
@article{Chen2026,
  author = {Chen, Houyuan and Li, Hong and Kong, Xianghao and Zhu, Tianrui and Xu, Shaocong and Xiao, Weiqing and Guo, Yuwei and Ye, Chongjie and Zhang, Lvmin and Zhao, Hao and Rao, Anyi},
  title = {UniVidX: A Unified Multimodal Framework for Versatile Video Generation via Diffusion Priors},
  journal = {ACM Transactions on Graphics},
  year = {2026},
  volume = {45},
  number = {4},
  pages = {1--17},
  doi = {10.1145/3811304}
}
Kostenlos starten →