Uni-Hand: Universal Hand Motion Forecasting in Egocentric Views
IEEE Transactions on Pattern Analysis and Machine Intelligence · 2026
DOI: 10.1109/tpami.2026.3692504 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2026. Verfasst von Junyi Ma und 7 weiteren Autor:innen. Hauptthemen laut MEDLINE: Hand, Image Processing, Computer-Assisted. Die Arbeit nennt 80 Literaturangaben. Gefördert von: National Key R&D Program of China, Explorers Program of Shanghai, Basic Research Funding.
Abstract (Originalsprache)
Forecasting how human hands move in egocentric views is critical for applications like augmented reality, human-robot policy transfer, and service/assistive technologies. Recently, several hand trajectory prediction (HTP) methods have been developed to generate future possible hand waypoints, which still suffer from insufficient prediction targets, inherent modality gaps, entangled hand-head motion, and limited validation in downstream tasks. To address these limitations, we present Uni-Hand, a universal hand motion forecasting framework considering multi-modal input, multi-dimensional and multi-target prediction patterns, and multi-task affordances for downstream applications. We harmonize multiple modalities by vision-language fusion, global context incorporation, and task-aware text embedding injection, to forecast hand waypoints in both 2D and 3D spaces. A novel dual-branch diffusion is proposed to concurrently predict human head and hand movements, capturing their motion synergy in egocentric vision. By introducing target indicators, the prediction model can forecast the specific joint waypoints of the wrist or the fingers, besides the widely studied hand center points. In addition, we enable Uni-Hand to additionally predict hand-object interaction states (contact/separation) to facilitate downstream tasks better. To incorporate comprehensive downstream task evaluations in the literature, we build novel benchmarks to assess the real-world applicability of hand motion forecasting algorithms. The experimental results on multiple publicly available datasets and our newly proposed benchmarks demonstrate that Uni-Hand achieves the state-of-the-art performance in multi-dimensional and multi-target hand motion forecasting. Extensive validation in multiple downstream tasks also presents its impressive human-robot policy transfer to enable robotic manipulation, and effective feature enhancement for action anticipation/recognition.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Förderung
- National Key R&D Program of China
- Explorers Program of Shanghai
- Basic Research Funding
- Shanghai Municipal Special Program for Basic Research
- Natural Science Foundation of China
Zitieren
APA
Ma, J., Bao, W., Xu, J., Sun, G., Zheng, Y., Zhang, E., Chen, X., & Wang, H. (2026). Uni-Hand: Universal Hand Motion Forecasting in Egocentric Views. IEEE Transactions on Pattern Analysis and Machine Intelligence. https://doi.org/10.1109/tpami.2026.3692504
Vancouver
Ma J, Bao W, Xu J, Sun G, Zheng Y, Zhang E, et al. Uni-Hand: Universal Hand Motion Forecasting in Egocentric Views. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2026. doi:10.1109/tpami.2026.3692504
BibTeX
@article{Ma2026,
author = {Ma, Junyi and Bao, Wentao and Xu, Jingyi and Sun, Guanzhong and Zheng, Yu and Zhang, Erhang and Chen, Xieyuanli and Wang, Hesheng},
title = {Uni-Hand: Universal Hand Motion Forecasting in Egocentric Views},
journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
year = {2026},
doi = {10.1109/tpami.2026.3692504}
}
EUVIMED