EUVIMEDEUVIMED entdecken
Crossref

Consistent 3D Hand Reconstruction in Video via Self-Supervised Learning

Zhigang Tu, Zhisheng Huang, Yujin Chen, Di Kang, Linchao Bao, Bisheng Yang, Junsong Yuan

IEEE Transactions on Pattern Analysis and Machine Intelligence · 2023-08 · 45 (8) · 9469-9485

DOI: 10.1109/tpami.2023.3247907 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023-08. Verfasst von Zhigang Tu und 6 weiteren Autor:innen. Hauptthemen laut MEDLINE: Algorithms, Benchmarking. Die Arbeit wurde bisher 83-mal zitiert (Crossref) und nennt 86 Literaturangaben. Gefördert von: Natural Science Fund for Distinguished Young Scholars of Hubei Province, National Natural Science Foundation of China, National Science Fund for Distinguished Young Scholars of China.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

We present a method for reconstructing accurate and consistent 3D hands from a monocular video. We observe that the detected 2D hand keypoints and the image texture provide important cues about the geometry and texture of the 3D hand, which can reduce or even eliminate the requirement on 3D hand annotation. Accordingly, in this work, we propose S2HAND, a self-supervised 3D hand reconstruction model, that can jointly estimate pose, shape, texture, and the camera viewpoint from a single RGB input through the supervision of easily accessible 2D detected keypoints. We leverage the continuous hand motion information contained in the unlabeled video data and explore S2HAND(V), which uses a set of weights shared S2HAND to process each frame and exploits additional motion, texture, and shape consistency constrains to obtain more accurate hand poses, and more consistent shapes and textures. Experiments on benchmark datasets demonstrate that our self-supervised method produces comparable hand reconstruction performance compared with the recent full-supervised methods in single-frame as input setup, and notably improves the reconstruction accuracy and consistency when using the video training data.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

AlgorithmsBenchmarkingCuesMotionSupervised Machine Learning

Förderung

  • Natural Science Fund for Distinguished Young Scholars of Hubei Province
  • National Natural Science Foundation of China
  • National Science Fund for Distinguished Young Scholars of China

Zitieren

APA

Tu, Z., Huang, Z., Chen, Y., Kang, D., Bao, L., Yang, B., & Yuan, J. (2023). Consistent 3D Hand Reconstruction in Video via Self-Supervised Learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(8), 9469-9485. https://doi.org/10.1109/tpami.2023.3247907

Vancouver

Tu Z, Huang Z, Chen Y, Kang D, Bao L, Yang B, et al. Consistent 3D Hand Reconstruction in Video via Self-Supervised Learning. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2023;45(8):9469-9485. doi:10.1109/tpami.2023.3247907

BibTeX
@article{Tu2023,
  author = {Tu, Zhigang and Huang, Zhisheng and Chen, Yujin and Kang, Di and Bao, Linchao and Yang, Bisheng and Yuan, Junsong},
  title = {Consistent 3D Hand Reconstruction in Video via Self-Supervised Learning},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2023},
  volume = {45},
  number = {8},
  pages = {9469--9485},
  doi = {10.1109/tpami.2023.3247907}
}
Kostenlos starten →