Learning realistic lip motions for humanoid face robots
Science Robotics · 2026
DOI: 10.1126/scirobotics.adx3017 ↗Kurz gefasst
Fachartikel in der Zeitschrift Science Robotics, 2026. Verfasst von Yuhang Hu und 14 weiteren Autor:innen. Beteiligte Einrichtung: Department of Mechanical Engineering, Columbia University, New York, NY, USA. Hauptthemen laut MEDLINE: Robotics, Lip. Die Arbeit wurde bisher 9-mal zitiert (Crossref) und nennt 51 Literaturangaben. Gefördert von: National Science Foundation.
Abstract (Originalsprache)
Lip motion represents outsized importance in human communication, capturing nearly half of our visual attention during conversation. Yet anthropomorphic robots often fail to achieve lip-audio synchronization, resulting in clumsy and lifeless lip behaviors. Two fundamental barriers underlay this challenge. First, robotic lips typically lack the mechanical complexity required to reproduce nuanced human mouth movements; second, existing synchronization methods depend on manually predefined movements and rules, restricting adaptability and realism. Here, we present a humanoid robot face designed to overcome these limitations, featuring soft silicone lips actuated by a 10–degree-of-freedom mechanism. To achieve lip synchronization without predefined movements, we used a self-supervised learning pipeline based on a variational autoencoder (VAE) combined with a facial action transformer, enabling the robot to autonomously infer more realistic lip trajectories directly from speech audio. Our experimental results suggest that this method outperforms simple heuristics like amplitude-based baselines in achieving more visually coherent lip-audio synchronization. Furthermore, the learned synchronization successfully generalizes across multiple linguistic contexts, enabling robot speech articulation in 10 languages unseen during training.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Autor:innen und Institutionen
- Department of Mechanical Engineering, Columbia University, New York, NY, USA.
- Department of Computer Science, Columbia University, New York, NY, USA.
- Department of Electrical Engineering, Columbia University, New York, NY, USA.
- Department of Mechanical Engineering & Materials Science, Duke University, Durham, NC, USA.
- Data Science Institute, Columbia University, New York, NY, USA.
ORCID-Profile: Yuhang Hu, Jiong Lin, Judah Allen Goldfeder, Philippe M Wyder, Yifeng Cao, Steven Tian, Yunzhe Wang, Cameron Mehlman, Yingke Wang, Delin Zeng, Boyuan Chen, Hod Lipson
Förderung
- National Science Foundation
Zitieren
APA
Hu, Y., Lin, J., Goldfeder, J. A., Wyder, P. M., Cao, Y., Tian, S., Wang, Y., Wang, J., Wang, M., Zeng, J., Mehlman, C., Wang, Y., Zeng, D., Chen, B., & Lipson, H. (2026). Learning realistic lip motions for humanoid face robots. Science Robotics. https://doi.org/10.1126/scirobotics.adx3017
Vancouver
Hu Y, Lin J, Goldfeder JA, Wyder PM, Cao Y, Tian S, et al. Learning realistic lip motions for humanoid face robots. Science Robotics. 2026. doi:10.1126/scirobotics.adx3017
BibTeX
@article{Hu2026,
author = {Hu, Yuhang and Lin, Jiong and Goldfeder, Judah Allen and Wyder, Philippe M and Cao, Yifeng and Tian, Steven and Wang, Yunzhe and Wang, Jingran and Wang, Mengmeng and Zeng, Jie and Mehlman, Cameron and Wang, Yingke and Zeng, Delin and Chen, Boyuan and Lipson, Hod},
title = {Learning realistic lip motions for humanoid face robots},
journal = {Science Robotics},
year = {2026},
doi = {10.1126/scirobotics.adx3017}
}
EUVIMED