EUVIMEDEUVIMED entdecken
Crossref

Manifold‐Aligned VQ Convolutional Adapters for Parameter‐Efficient Cross‐Lingual Speech‐Driven 3D Facial Animation

YuGuang Jin, Hui Luo

Computer Graphics Forum · 04.11.2026

DOI: 10.1111/cgf.70575 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Computer Graphics Forum, online erschienen am 04.11.2026. Verfasst von YuGuang Jin und 1 weiteren Autor:innen. Die Arbeit nennt 29 Literaturangaben. Gefördert von: Natural Science Foundation of Heilongjiang Province, National Natural Science Foundation of China.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Speech‐driven 3D facial animation has achieved remarkable realism, yet existing models suffer from severe performance degradation on unseen languages due to discrepancies in articulation patterns and motion distributions. We attribute this limitation to a distribution shift on the latent motion manifold, where pretrained vector‐quantized (VQ) codebooks fail to adequately cover target‐language articulation regions. To address this, we propose Manifold‐Aligned VQ Convolutional Adapters for parameter‐efficient cross‐lingual adaptation. Instead of retraining the entire backbone, we insert lightweight 1D convolutional adapters into the post‐quantization and decoder stages, formulating adaptation as a localized geometric correction on the motion manifold. Theoretically, we demonstrate that under the local bi‐Lipschitz continuity of the pretrained decoder, minimizing reconstruction loss effectively aligns the latent motion distribution with the target language. Unlike conventional linear adapters, our convolutional design explicitly captures local temporal dependencies and higher‐order motion dynamics critical for language‐specific articulation. To support systematic evaluation, we construct CMLR‐3D, a paired Mandarin speech–3D facial motion dataset. Extensive experiments on CodeTalker and ARTalk show that our method significantly improves lip synchronization and motion naturalness, consistently outperforming full and partial fine‐tuning baselines in both quantitative metrics (LVE, AVLR) and perceptual user studies. Notably, the framework updates only 0.46%–1.26% of parameters while preserving pretrained motion priors. This work provides a theoretically grounded, highly efficient paradigm for multilingual virtual avatar deployment. (Anonymous link http:github.com/GreenTomato‐hugo/MAVQ‐Adapter )

Förderung

  • Natural Science Foundation of Heilongjiang Province
  • National Natural Science Foundation of China

Zitieren

APA

Jin, Y., & Luo, H. (2026). Manifold‐Aligned VQ Convolutional Adapters for Parameter‐Efficient Cross‐Lingual Speech‐Driven 3D Facial Animation. Computer Graphics Forum. https://doi.org/10.1111/cgf.70575

Vancouver

Jin Y, Luo H. Manifold‐Aligned VQ Convolutional Adapters for Parameter‐Efficient Cross‐Lingual Speech‐Driven 3D Facial Animation. Computer Graphics Forum. 2026. doi:10.1111/cgf.70575

BibTeX
@article{Jin2026,
  author = {Jin, YuGuang and Luo, Hui},
  title = {Manifold‐Aligned VQ Convolutional Adapters for Parameter‐Efficient Cross‐Lingual Speech‐Driven 3D Facial Animation},
  journal = {Computer Graphics Forum},
  year = {2026},
  doi = {10.1111/cgf.70575}
}
Kostenlos starten →