EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC-ND 4.0

HimaTalk: robust audio-driven Tibetan talking head generation via bi-directional Mamba and spatiotemporal constraints

Tingting Du, Xiangzhen He, Xianghe Meng, Xue Ding, Wenhao Jiang, Xiang Wang, Haotian Zhu, Jingbin Hao, Shibiao Shi, Xu Ouyang, Ruilei Shan

Journal of King Saud University Computer and Information Sciences · 2026-07 · 38 (5)

DOI: 10.1007/s44443-026-00617-6 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of King Saud University Computer and Information Sciences, online erschienen am 04.03.2026. Verfasst von Tingting Du und 10 weiteren Autor:innen. Die Arbeit nennt 44 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0). Gefördert von: National Natural Science Foundation of China, Central Government Guide Local Science and Technology Development Fund of Gansu Province, Science and Technology Program of Lanzhou City.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

High-fidelity audio-driven talking-head generation holds significant promise for digital communication, online education, and the preservation of minority languages. However, applying existing diffusion-based frameworks to low-resource languages like Tibetan remains challenging, primarily due to data scarcity and the difficulty of generalizing to unseen prosodic patterns. Current state-of-the-art systems typically rely on Transformer-based temporal attention mechanisms. While effective for short sequences, these architectures suffer from quadratic complexity (O(T2)\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {O}(T^2)$$\end{document}), which limits the context window and hinders the modeling of long-range temporal dependencies. In cross-lingual transfer, these limitations can be further exacerbated, and models may struggle to maintain structural stability, leading to artifacts such as motion stiffness, background warping, and temporal flickering. To address these limitations, we present HimaTalk (Himalayan Talking Head), a unified framework designed to optimize generation quality, computational efficiency, and cross-lingual generalization. We replace the standard temporal attention layer with a Bidirectional Mamba Motion Module. By employing a dual scanning mechanism (forward and backward) with learnable gated fusion, this module captures global temporal context with linear complexity (O(T)\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {O}(T)$$\end{document}), enabling the model to incorporate both past and future audio cues for smoother transitions. Furthermore, to ensure robustness against domain shifts, we introduce a spatiotemporal dual-constraint strategy: a facial region-weighted spatial loss to effectively decouple foreground dynamics from the background, and a latent second-order difference consistency loss to suppress high-frequency jitter. We also release the Tibetan Talking Head Dataset (TTHD), the first high-definition talking-head video dataset for Tibetan, comprising approximately 40 hours of video with diverse emotional expressions. Extensive experiments demonstrate that HimaTalk achieves state-of-the-art zero-shot performance on TTHD and competitive results on a comparable real-world dataset. Notably, our framework reduces peak memory usage by 35% and increases inference speed by approximately 45% compared to temporal-attention baselines under comparable settings, while delivering superior temporal coherence to unidirectional state-space model (SSM) approaches.

Förderung

  • National Natural Science Foundation of China
  • Central Government Guide Local Science and Technology Development Fund of Gansu Province
  • Science and Technology Program of Lanzhou City
  • Education and Teaching Achievement Cultivation Project of Gansu Province

Zitieren

APA

Du, T., He, X., Meng, X., Ding, X., Jiang, W., Wang, X., Zhu, H., Hao, J., Shi, S., Ouyang, X., & Shan, R. (2026). HimaTalk: robust audio-driven Tibetan talking head generation via bi-directional Mamba and spatiotemporal constraints. Journal of King Saud University Computer and Information Sciences, 38(5). https://doi.org/10.1007/s44443-026-00617-6

Vancouver

Du T, He X, Meng X, Ding X, Jiang W, Wang X, et al. HimaTalk: robust audio-driven Tibetan talking head generation via bi-directional Mamba and spatiotemporal constraints. Journal of King Saud University Computer and Information Sciences. 2026;38(5). doi:10.1007/s44443-026-00617-6

BibTeX
@article{Du2026,
  author = {Du, Tingting and He, Xiangzhen and Meng, Xianghe and Ding, Xue and Jiang, Wenhao and Wang, Xiang and Zhu, Haotian and Hao, Jingbin and Shi, Shibiao and Ouyang, Xu and Shan, Ruilei},
  title = {HimaTalk: robust audio-driven Tibetan talking head generation via bi-directional Mamba and spatiotemporal constraints},
  journal = {Journal of King Saud University Computer and Information Sciences},
  year = {2026},
  volume = {38},
  number = {5},
  doi = {10.1007/s44443-026-00617-6}
}
Kostenlos starten →