CAGNet: a context-aware graph neural network for detecting social relationships in videos
Visual Intelligence · 23.07.2024 · 2 (1)
DOI: 10.1007/s44267-024-00056-9 ↗Kurz gefasst
Fachartikel in der Zeitschrift Visual Intelligence, online erschienen am 23.07.2024. Verfasst von Fan Yu und 5 weiteren Autor:innen. Die Arbeit wurde bisher 10-mal zitiert (Crossref) und nennt 59 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Natural Science Foundation of China, the Fundamental Research Funds for the Central Universities, the Collaborative Innovation Center of Novel Software Technology and Industrialization.
Abstract (Originalsprache)
Social relationships, such as parent-offspring and friends, are crucial and stable connections between individuals, especially at the person level, and are essential for accurately describing the semantics of videos. In this paper, we analogize such a task to scene graph generation, which we call video social relationship graph generation (VSRGG). It involves generating a social relationship graph for each video based on person-level relationships. We propose a context-aware graph neural network (CAGNet) for VSRGG, which effectively generates social relationship graphs through message passing, capturing the context of the video. Specifically, CAGNet detects persons in the video, generates an initial graph via relationship proposal, and extracts facial and body features to describe the detected individuals, as well as temporal features to describe their interactions. Then, CAGNet predicts pairwise relationships between individuals using graph message passing. Additionally, we construct a new dataset, VidSoR, to evaluate VSRGG, which contains 72 h of video with 6276 person instances and 5313 relationship instances of eight relationship types. Extensive experiments show that CAGNet can make accurate predictions with a comparatively high mean recall (mRecall) when using only visual features.
Förderung
- National Natural Science Foundation of China
- the Fundamental Research Funds for the Central Universities
- the Collaborative Innovation Center of Novel Software Technology and Industrialization
Zitieren
APA
Yu, F., Fang, Y., Zhao, Z., Bei, J., Ren, T., & Wu, G. (2024). CAGNet: a context-aware graph neural network for detecting social relationships in videos. Visual Intelligence, 2(1). https://doi.org/10.1007/s44267-024-00056-9
Vancouver
Yu F, Fang Y, Zhao Z, Bei J, Ren T, Wu G. CAGNet: a context-aware graph neural network for detecting social relationships in videos. Visual Intelligence. 2024;2(1). doi:10.1007/s44267-024-00056-9
BibTeX
@article{Yu2024,
author = {Yu, Fan and Fang, Yaqun and Zhao, Zhixiang and Bei, Jia and Ren, Tongwei and Wu, Gangshan},
title = {CAGNet: a context-aware graph neural network for detecting social relationships in videos},
journal = {Visual Intelligence},
year = {2024},
volume = {2},
number = {1},
doi = {10.1007/s44267-024-00056-9}
}
EUVIMED