EUVIMEDEUVIMED entdecken
Crossref

Dubbing Movies via Hierarchical Phoneme Modeling and Acoustic Diffusion Denoising

Liang Li, Gaoxiang Cong, Yuankai Qi, Zheng-Jun Zha, Qi Wu, Quan Z Sheng, Qingming Huang, Ming-Hsuan Yang

IEEE Transactions on Pattern Analysis and Machine Intelligence · 2025-11 · 47 (11) · 10361-10377

DOI: 10.1109/tpami.2025.3597267 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025-11. Verfasst von Liang Li und 7 weiteren Autor:innen. Hauptthemen laut MEDLINE: Video Recording, Image Processing, Computer-Assisted, Phonetics. Die Arbeit wurde bisher 10-mal zitiert (Crossref) und nennt 81 Literaturangaben.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Given a piece of text, a video clip, and reference audio, the movie dubbing (also known as Visual Voice Cloning, V2C) task aims to generate speeches that clone reference voice and align well with the video in both emotion and lip movement, which is more challenging than conventional text-to-speech synthesis tasks. To align the generated speech with the inherent lip motion of the given silent video, most existing works utilize each video frame to query textual phonemes. However, such an attention operation usually leads to mumble speech because different phonemes are fused for video frames corresponding to one phoneme (video frames are finer-grained than phonemes). To address this issue, we propose a diffusion-based movie dubbing architecture, which improves pronunciation by Hierarchical Phoneme Modeling (HPM) and generates better mel-spectrogram through Acoustic Diffusion Denoising (ADD). We term our model as HD-Dubber. Specifically, our HPM bridges the visual information and corresponding speech prosody from three aspects: (1) aligning lip movement with the speech duration based on each phoneme unit by contrastive learning; (2) conveying facial expression to phoneme-level energy and pitch; and (3) injecting global emotions captured from video scenes into prosody. On the other hand, ADD exploits a denoising diffusion framework to transform the noise signal into a mel-spectrogram via a parameterized Markov chain conditioned on textual phonemes and reference audio. ADD has two novel denoisers, the Style-adaptive Residual Denoiser (SRD) and the Phoneme-enhanced U-net Denoiser (PUD), to enhance speaker similarity and improve pronunciation quality. Extensive experimental results on the three benchmark datasets demonstrate the state-of-the-art performance of the proposed method. The source code and trained models will be made available to the public.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Video RecordingImage Processing, Computer-AssistedPhoneticsHumansAlgorithmsSpeechMotion Pictures

Zitieren

APA

Li, L., Cong, G., Qi, Y., Zha, Z. J., Wu, Q., Sheng, Q. Z., Huang, Q., & Yang, M. H. (2025). Dubbing Movies via Hierarchical Phoneme Modeling and Acoustic Diffusion Denoising. IEEE Transactions on Pattern Analysis and Machine Intelligence, 47(11), 10361-10377. https://doi.org/10.1109/tpami.2025.3597267

Vancouver

Li L, Cong G, Qi Y, Zha ZJ, Wu Q, Sheng QZ, et al. Dubbing Movies via Hierarchical Phoneme Modeling and Acoustic Diffusion Denoising. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2025;47(11):10361-10377. doi:10.1109/tpami.2025.3597267

BibTeX
@article{Li2025,
  author = {Li, Liang and Cong, Gaoxiang and Qi, Yuankai and Zha, Zheng-Jun and Wu, Qi and Sheng, Quan Z and Huang, Qingming and Yang, Ming-Hsuan},
  title = {Dubbing Movies via Hierarchical Phoneme Modeling and Acoustic Diffusion Denoising},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2025},
  volume = {47},
  number = {11},
  pages = {10361--10377},
  doi = {10.1109/tpami.2025.3597267}
}
Kostenlos starten →