ViTPose++: Vision Transformer for Generic Body Pose Estimation
IEEE Transactions on Pattern Analysis and Machine Intelligence · 2024-02 · 46 (2) · 1212-1230
DOI: 10.1109/tpami.2023.3330016 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024-02. Verfasst von Yufei Xu und 3 weiteren Autor:innen. Die Arbeit wurde bisher 162-mal zitiert (Crossref) und nennt 84 Literaturangaben. Gefördert von: Australian Research Council.
Abstract (Originalsprache)
In this paper, we show the surprisingly good properties of plain vision transformers for body pose estimation from various aspects, namely simplicity in model structure, scalability in model size, flexibility in training paradigm, and transferability of knowledge between models, through a simple baseline model dubbed ViTPose. ViTPose employs the plain and non-hierarchical vision transformer as an encoder to encode features and a lightweight decoder to decode body keypoints in either a top-down or a bottom-up manner. It can be scaled to 1B parameters by taking the advantage of the scalable model capacity and high parallelism, setting a new Pareto front for throughput and performance. Besides, ViTPose is very flexible regarding the attention type, input resolution, and pre-training and fine-tuning strategy. Based on the flexibility, a novel ViTPose++ model is proposed to deal with heterogeneous body keypoint categories via knowledge factorization, i.e., adopting task-agnostic and task-specific feed-forward networks in the transformer. We also demonstrate that the knowledge of large ViTPose models can be easily transferred to small ones via a simple knowledge token. Our largest single model ViTPose-G sets a new record on the MS COCO test set without model ensemble. Furthermore, our ViTPose++ model achieves state-of-the-art performance simultaneously on a series of body pose estimation tasks, including MS COCO, AI Challenger, OCHuman, MPII for human keypoint detection, COCO-Wholebody for whole-body keypoint detection, as well as AP-10K and APT-36K for animal keypoint detection, without sacrificing inference speed.
Förderung
- Australian Research Council
Zitieren
APA
Xu, Y., Zhang, J., Zhang, Q., & Tao, D. (2024). ViTPose++: Vision Transformer for Generic Body Pose Estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(2), 1212-1230. https://doi.org/10.1109/tpami.2023.3330016
Vancouver
Xu Y, Zhang J, Zhang Q, Tao D. ViTPose++: Vision Transformer for Generic Body Pose Estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2024;46(2):1212-1230. doi:10.1109/tpami.2023.3330016
BibTeX
@article{Xu2024,
author = {Xu, Yufei and Zhang, Jing and Zhang, Qiming and Tao, Dacheng},
title = {ViTPose++: Vision Transformer for Generic Body Pose Estimation},
journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
year = {2024},
volume = {46},
number = {2},
pages = {1212--1230},
doi = {10.1109/tpami.2023.3330016}
}
EUVIMED