EUVIMEDEUVIMED entdecken
Crossref

Free-Form Composition Networks for Egocentric Action Recognition

Haoran Wang, Qinghua Cheng, Baosheng Yu, Yibing Zhan, Dapeng Tao, Liang Ding, Haibin Ling

IEEE Transactions on Circuits and Systems for Video Technology · 2024-10 · 34 (10) · 9967-9978

DOI: 10.1109/tcsvt.2024.3406546 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Circuits and Systems for Video Technology, 2024-10. Verfasst von Haoran Wang und 6 weiteren Autor:innen. Die Arbeit wurde bisher 5-mal zitiert (Crossref) und nennt 84 Literaturangaben. Gefördert von: Major Science and Technology Innovation 2030 “New Generation Artificial Intelligence” Key Project, Yunnan Dengcheng Expert Workstation, Yunnan Key Laboratory of Media Convergence.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Egocentric action recognition is gaining significant attention in the field of human action recognition. In this paper, we address data scarcity issue in egocentric action recognition from a compositional generalization perspective. To tackle this problem, we propose a free-form composition network (FFCN) that can simultaneously learn disentangled verb, preposition, and noun representations, and then use them to compose new samples in the feature space for rare classes of action videos. First, we use a graph to capture the spatial-temporal relations among different hand/object instances in each action video. We thus decompose each action into a set of verb and preposition spatial-temporal representations using the edge features in the graph. The temporal decomposition extracts verb and preposition representations from different video frames, while the spatial decomposition adaptively learns verb and preposition representations from action-related instances in each frame. With these spatial-temporal representations of verbs and prepositions, we can compose new samples for those rare classes in a free-form manner, which is not restricted to a rigid form of a verb and a noun. The proposed FFCN can directly generate new training data samples for rare classes, hence significantly improve action recognition performance. We evaluated our method on three popular egocentric action recognition datasets, Something-Something V2, H2O, and EPIC-KITCHENS-100, and the experimental results demonstrate the effectiveness of the proposed method for handling data scarcity problems, including long-tailed and few-shot egocentric action recognition.

Förderung

  • Major Science and Technology Innovation 2030 “New Generation Artificial Intelligence” Key Project
  • Yunnan Dengcheng Expert Workstation
  • Yunnan Key Laboratory of Media Convergence
  • Fundamental Research Funds for the Central Universities of China
  • Research Fund of Liaoning Province

Zitieren

APA

Wang, H., Cheng, Q., Yu, B., Zhan, Y., Tao, D., Ding, L., & Ling, H. (2024). Free-Form Composition Networks for Egocentric Action Recognition. IEEE Transactions on Circuits and Systems for Video Technology, 34(10), 9967-9978. https://doi.org/10.1109/tcsvt.2024.3406546

Vancouver

Wang H, Cheng Q, Yu B, Zhan Y, Tao D, Ding L, et al. Free-Form Composition Networks for Egocentric Action Recognition. IEEE Transactions on Circuits and Systems for Video Technology. 2024;34(10):9967-9978. doi:10.1109/tcsvt.2024.3406546

BibTeX
@article{Wang2024,
  author = {Wang, Haoran and Cheng, Qinghua and Yu, Baosheng and Zhan, Yibing and Tao, Dapeng and Ding, Liang and Ling, Haibin},
  title = {Free-Form Composition Networks for Egocentric Action Recognition},
  journal = {IEEE Transactions on Circuits and Systems for Video Technology},
  year = {2024},
  volume = {34},
  number = {10},
  pages = {9967--9978},
  doi = {10.1109/tcsvt.2024.3406546}
}
Kostenlos starten →