EUVIMEDEUVIMED entdecken
Crossref

ViTPose++: Vision Transformer for Generic Body Pose Estimation

Yufei Xu, Jing Zhang, Qiming Zhang, Dacheng Tao

IEEE Transactions on Pattern Analysis and Machine Intelligence · 2024-02 · 46 (2) · 1212-1230

DOI: 10.1109/tpami.2023.3330016 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024-02. Verfasst von Yufei Xu und 3 weiteren Autor:innen. Die Arbeit wurde bisher 162-mal zitiert (Crossref) und nennt 84 Literaturangaben. Gefördert von: Australian Research Council.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

In this paper, we show the surprisingly good properties of plain vision transformers for body pose estimation from various aspects, namely simplicity in model structure, scalability in model size, flexibility in training paradigm, and transferability of knowledge between models, through a simple baseline model dubbed ViTPose. ViTPose employs the plain and non-hierarchical vision transformer as an encoder to encode features and a lightweight decoder to decode body keypoints in either a top-down or a bottom-up manner. It can be scaled to 1B parameters by taking the advantage of the scalable model capacity and high parallelism, setting a new Pareto front for throughput and performance. Besides, ViTPose is very flexible regarding the attention type, input resolution, and pre-training and fine-tuning strategy. Based on the flexibility, a novel ViTPose++ model is proposed to deal with heterogeneous body keypoint categories via knowledge factorization, i.e., adopting task-agnostic and task-specific feed-forward networks in the transformer. We also demonstrate that the knowledge of large ViTPose models can be easily transferred to small ones via a simple knowledge token. Our largest single model ViTPose-G sets a new record on the MS COCO test set without model ensemble. Furthermore, our ViTPose++ model achieves state-of-the-art performance simultaneously on a series of body pose estimation tasks, including MS COCO, AI Challenger, OCHuman, MPII for human keypoint detection, COCO-Wholebody for whole-body keypoint detection, as well as AP-10K and APT-36K for animal keypoint detection, without sacrificing inference speed.

Förderung

  • Australian Research Council

Zitieren

APA

Xu, Y., Zhang, J., Zhang, Q., & Tao, D. (2024). ViTPose++: Vision Transformer for Generic Body Pose Estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(2), 1212-1230. https://doi.org/10.1109/tpami.2023.3330016

Vancouver

Xu Y, Zhang J, Zhang Q, Tao D. ViTPose++: Vision Transformer for Generic Body Pose Estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2024;46(2):1212-1230. doi:10.1109/tpami.2023.3330016

BibTeX
@article{Xu2024,
  author = {Xu, Yufei and Zhang, Jing and Zhang, Qiming and Tao, Dacheng},
  title = {ViTPose++: Vision Transformer for Generic Body Pose Estimation},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2024},
  volume = {46},
  number = {2},
  pages = {1212--1230},
  doi = {10.1109/tpami.2023.3330016}
}
Kostenlos starten →