EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Matching Compound Prototypes for Few-Shot Action Recognition

Yifei Huang, Lijin Yang, Guo Chen, Hongjie Zhang, Feng Lu, Yoichi Sato

International Journal of Computer Vision · 2024-09 · 132 (9) · 3977-4002

DOI: 10.1007/s11263-024-02017-7 ↗

Kurz gefasst

Fachartikel in der Zeitschrift International Journal of Computer Vision, online erschienen am 29.04.2024. Verfasst von Yifei Huang und 5 weiteren Autor:innen. Die Arbeit wurde bisher 22-mal zitiert (Crossref) und nennt 91 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Japan Society for the Promotion of Science, Japan Science and Technology Agency.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The task of few-shot action recognition aims to recognize novel action classes using only a small number of labeled training samples. How to better describe the action in each video and how to compare the similarity between videos are two of the most critical factors in this task. Directly describing the video globally or by its individual frames cannot well represent the spatiotemporal dependencies within an action. On the other hand, naively matching the global representations of two videos is also not optimal since action can happen at different locations in a video with different speeds. In this work, we propose a novel approach that describes each video using multiple types of prototypes and then computes the video similarity with a particular matching strategy for each type of prototypes. To better model the spatiotemporal dependency, we describe the video by generating prototypes that model the multi-level spatiotemporal relations via transformers. There are a total of three types of prototypes. The first type of prototypes are trained to describe specific aspects of the action in the video e.g., the start of the action, regardless of its timestamp. These prototypes are directly matched one-to-one between two videos to compare their similarity. The second type of prototypes are the timestamp-centered prototypes that are trained to focus on specific timestamps of the video. To deal with the temporal variation of actions in a video, we apply bipartite matching to allow the matching of prototypes of different timestamps. The third type of prototypes are generated from the timestamp-centered prototypes, which regularize their temporal consistency while serving as an auxiliary summarization of the whole video. Experiments demonstrate that our proposed method achieves state-of-the-art results on multiple benchmarks.

Förderung

  • Japan Society for the Promotion of Science
  • Japan Science and Technology Agency

Zitieren

APA

Huang, Y., Yang, L., Chen, G., Zhang, H., Lu, F., & Sato, Y. (2024). Matching Compound Prototypes for Few-Shot Action Recognition. International Journal of Computer Vision, 132(9), 3977-4002. https://doi.org/10.1007/s11263-024-02017-7

Vancouver

Huang Y, Yang L, Chen G, Zhang H, Lu F, Sato Y. Matching Compound Prototypes for Few-Shot Action Recognition. International Journal of Computer Vision. 2024;132(9):3977-4002. doi:10.1007/s11263-024-02017-7

BibTeX
@article{Huang2024,
  author = {Huang, Yifei and Yang, Lijin and Chen, Guo and Zhang, Hongjie and Lu, Feng and Sato, Yoichi},
  title = {Matching Compound Prototypes for Few-Shot Action Recognition},
  journal = {International Journal of Computer Vision},
  year = {2024},
  volume = {132},
  number = {9},
  pages = {3977--4002},
  doi = {10.1007/s11263-024-02017-7}
}
Kostenlos starten →