EUVIMEDEUVIMED entdecken
Crossref

Playing for 3D Human Recovery

Zhongang Cai, Mingyuan Zhang, Jiawei Ren, Chen Wei, Daxuan Ren, Zhengyu Lin, Haiyu Zhao, Lei Yang, Chen Change Loy, Ziwei Liu

IEEE Transactions on Pattern Analysis and Machine Intelligence · 2024

DOI: 10.1109/tpami.2024.3450537 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024. Verfasst von Zhongang Cai und 9 weiteren Autor:innen. Hauptthemen laut MEDLINE: Imaging, Three-Dimensional, Video Games, Video Recording. Die Arbeit wurde bisher 22-mal zitiert (Crossref) und nennt 76 Literaturangaben. Gefördert von: Ministry of Education - Singapore, MOE AcRF Tier 2, NTU NAP.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Image- and video-based 3D human recovery (i.e., pose and shape estimation) have achieved substantial progress. However, due to the prohibitive cost of motion capture, existing datasets are often limited in scale and diversity. In this work, we obtain massive human sequences by playing the video game with automatically annotated 3D ground truths. Specifically, we contribute GTA-Human, a large-scale 3D human dataset generated with the GTA-V game engine, featuring a highly diverse set of subjects, actions, and scenarios. More importantly, we study the use of game-playing data and obtain five major insights. First, game-playing data is surprisingly effective. A simple frame-based baseline trained on GTA-Human outperforms more sophisticated methods by a large margin. For video-based methods, GTA-Human is even on par with the in-domain training set. Second, we discover that synthetic data provides critical complements to the real data that is typically collected indoor. We highlight that our investigation into domain gap provides explanations for our data mixture strategies that are simple yet useful, which offers new insights to the research community. Third, the scale of the dataset matters. The performance boost is closely related to the additional data available. A systematic study on multiple key factors (such as camera angle and body pose) reveals that the model performance is sensitive to data density. Fourth, the effectiveness of GTA-Human is also attributed to the rich collection of strong supervision labels (SMPL parameters), which are otherwise expensive to acquire in real datasets. Fifth, the benefits of synthetic data extend to larger models such as deeper convolutional neural networks (CNNs) and Transformers, for which a significant impact is also observed. We hope our work could pave the way for scaling up 3D human recovery to the real world.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Imaging, Three-DimensionalVideo GamesVideo RecordingHumansPostureAlgorithmsDatabases, Factual

Förderung

  • Ministry of Education - Singapore
  • MOE AcRF Tier 2
  • NTU NAP
  • RIE2020 Industry Alignment Fund - Industry Collaboration Projects

Zitieren

APA

Cai, Z., Zhang, M., Ren, J., Wei, C., Ren, D., Lin, Z., Zhao, H., Yang, L., Loy, C. C., & Liu, Z. (2024). Playing for 3D Human Recovery. IEEE Transactions on Pattern Analysis and Machine Intelligence. https://doi.org/10.1109/tpami.2024.3450537

Vancouver

Cai Z, Zhang M, Ren J, Wei C, Ren D, Lin Z, et al. Playing for 3D Human Recovery. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2024. doi:10.1109/tpami.2024.3450537

BibTeX
@article{Cai2024,
  author = {Cai, Zhongang and Zhang, Mingyuan and Ren, Jiawei and Wei, Chen and Ren, Daxuan and Lin, Zhengyu and Zhao, Haiyu and Yang, Lei and Loy, Chen Change and Liu, Ziwei},
  title = {Playing for 3D Human Recovery},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2024},
  doi = {10.1109/tpami.2024.3450537}
}
Kostenlos starten →