EUVIMEDEUVIMED entdecken
Crossref

Modular-Cam: Modular Dynamic Camera-view Video Generation with LLM

Zirui Pan, Xin Wang, Yipeng Zhang, Hong Chen, Kwan Man Cheng, Yaofei Wu, Wenwu Zhu

Proceedings of the AAAI Conference on Artificial Intelligence · 11.04.2025 · 39 (6) · 6363-6371

DOI: 10.1609/aaai.v39i6.32681 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 11.04.2025. Verfasst von Zirui Pan und 6 weiteren Autor:innen. Die Arbeit wurde bisher 3-mal zitiert (Crossref).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Text-to-Video generation, which utilizes the provided text prompt to generate high-quality videos, has drawn increasing attention and achieved great success due to the development of diffusion models recently. Existing methods mainly rely on a pre-trained text encoder to capture the semantic information and perform cross attention with the encoded text prompt to guide the generation of video. However, when it comes to complex prompts that contain dynamic scenes and multiple camera-view transformations, these methods can not decompose the overall information into separate scenes, as well as fail to smoothly change scenes based on the corresponding camera-views. To solve these problems, we propose a novel method, i.e., Modular-Cam. Specifically, to better understand a given complex prompt, we utilize a large language model to analyze user instructions and decouple them into multiple scenes together with transition actions. To generate a video containing dynamic scenes that match the given camera-views, we incorporate the widely-used temporal transformer into the diffusion model to ensure continuity within a single scene and propose CamOperator, a modular network based module that well controls the camera movements. Moreover, we propose AdaControlNet, which utilizes ControlNet to ensure consistency across scenes and adaptively adjusts the color tone of the generated video. Extensive qualitative and quantitative experiments prove our proposed Modular-Cam's strong capability of generating multi-scene videos together with its ability to achieve fine-grained control of camera movements. Generated results are available at https://modular-cam.github.io.

Zitieren

APA

Pan, Z., Wang, X., Zhang, Y., Chen, H., Cheng, K. M., Wu, Y., & Zhu, W. (2025). Modular-Cam: Modular Dynamic Camera-view Video Generation with LLM. Proceedings of the AAAI Conference on Artificial Intelligence, 39(6), 6363-6371. https://doi.org/10.1609/aaai.v39i6.32681

Vancouver

Pan Z, Wang X, Zhang Y, Chen H, Cheng KM, Wu Y, et al. Modular-Cam: Modular Dynamic Camera-view Video Generation with LLM. Proceedings of the AAAI Conference on Artificial Intelligence. 2025;39(6):6363-6371. doi:10.1609/aaai.v39i6.32681

BibTeX
@article{Pan2025,
  author = {Pan, Zirui and Wang, Xin and Zhang, Yipeng and Chen, Hong and Cheng, Kwan Man and Wu, Yaofei and Zhu, Wenwu},
  title = {Modular-Cam: Modular Dynamic Camera-view Video Generation with LLM},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2025},
  volume = {39},
  number = {6},
  pages = {6363--6371},
  doi = {10.1609/aaai.v39i6.32681}
}
Kostenlos starten →