EUVIMEDEUVIMED entdecken
Crossref

AV-GEN: TWIN BACKBONE CROSS-MODAL FUSION FOR AUDIO-VIDEO GENERATION

Subha Mishra, Naveen Singh, Nalin Tiwari

INTERNATIONAL JOURNAL OF SCIENTIFIC RESEARCH IN ENGINEERING AND MANAGEMENT · 09.04.2026 · 10 (04) · 1-9

DOI: 10.55041/ijsrem59037 ↗

Kurz gefasst

Fachartikel in der Zeitschrift INTERNATIONAL JOURNAL OF SCIENTIFIC RESEARCH IN ENGINEERING AND MANAGEMENT, online erschienen am 09.04.2026. Verfasst von Subha Mishra und 2 weiteren Autor:innen.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Absract Audio–video (AV) generation has often relied on complex multi-stage architec- tures or sequential synthesis of sound and visuals. We introduce OVI, a unified paradigm for audio–video generation that models the two modalities as a single generative process. By using blockwise cross-modal fusion of twin-DiT modules, OVI achieves natural synchronization and removes the need for separate pipelines or post hoc alignment. To facilitate fine-grained multimodal fusion modeling, we initialize an audio tower with an architecture identical to that of a strong pretrained video model. Trained from scratch on hundreds of thousands of hours of raw au- dio, the audio tower learns to generate realistic sound effects, as well as speech that conveys rich speaker identity and emotion. Fusion is obtained by jointly training the identical video and audio towers via blockwise exchange of timing (via scaled- RoPE embeddings) and semantics (through bidirectional cross-attention) on a vast video corpus. Our model enables cinematic storytelling with natural speech and accurate, context-matched sound effects, producing movie-grade video clips.

Zitieren

APA

Mishra, S., Singh, N., & Tiwari, N. (2026). AV-GEN: TWIN BACKBONE CROSS-MODAL FUSION FOR AUDIO-VIDEO GENERATION. INTERNATIONAL JOURNAL OF SCIENTIFIC RESEARCH IN ENGINEERING AND MANAGEMENT, 10(04), 1-9. https://doi.org/10.55041/ijsrem59037

Vancouver

Mishra S, Singh N, Tiwari N. AV-GEN: TWIN BACKBONE CROSS-MODAL FUSION FOR AUDIO-VIDEO GENERATION. INTERNATIONAL JOURNAL OF SCIENTIFIC RESEARCH IN ENGINEERING AND MANAGEMENT. 2026;10(04):1-9. doi:10.55041/ijsrem59037

BibTeX
@article{Mishra2026,
  author = {Mishra, Subha and Singh, Naveen and Tiwari, Nalin},
  title = {AV-GEN: TWIN BACKBONE CROSS-MODAL FUSION FOR AUDIO-VIDEO GENERATION},
  journal = {INTERNATIONAL JOURNAL OF SCIENTIFIC RESEARCH IN ENGINEERING AND MANAGEMENT},
  year = {2026},
  volume = {10},
  number = {04},
  pages = {1--9},
  doi = {10.55041/ijsrem59037}
}
Kostenlos starten →