End-to-End Zero-Shot HOI Detection via Vision and Language Knowledge Distillation
Proceedings of the AAAI Conference on Artificial Intelligence · 26.06.2023 · 37 (3) · 2839-2846
DOI: 10.1609/aaai.v37i3.25385 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 26.06.2023. Verfasst von Mingrui Wu und 5 weiteren Autor:innen. Die Arbeit wurde bisher 42-mal zitiert (Crossref).
Abstract (Originalsprache)
Most existing Human-Object Interaction (HOI) Detection methods rely heavily on full annotations with predefined HOI categories, which is limited in diversity and costly to scale further. We aim at advancing zero-shot HOI detection to detect both seen and unseen HOIs simultaneously. The fundamental challenges are to discover potential human-object pairs and identify novel HOI categories. To overcome the above challenges, we propose a novel End-to-end zero-shot HOI Detection (EoID) framework via vision-language knowledge distillation. We first design an Interactive Score module combined with a Two-stage Bipartite Matching algorithm to achieve interaction distinguishment for human-object pairs in an action-agnostic manner. Then we transfer the distribution of action probability from the pretrained vision-language teacher as well as the seen ground truth to the HOI model to attain zero-shot HOI classification. Extensive experiments on HICO-Det dataset demonstrate that our model discovers potential interactive pairs and enables the recognition of unseen HOIs. Finally, our method outperforms the previous SOTA under various zero-shot settings. Moreover, our method is generalizable to large-scale object detection data to further scale up the action sets. The source code is available at: https://github.com/mrwu-mac/EoID.
Zitieren
APA
Wu, M., Gu, J., Shen, Y., Lin, M., Chen, C., & Sun, X. (2023). End-to-End Zero-Shot HOI Detection via Vision and Language Knowledge Distillation. Proceedings of the AAAI Conference on Artificial Intelligence, 37(3), 2839-2846. https://doi.org/10.1609/aaai.v37i3.25385
Vancouver
Wu M, Gu J, Shen Y, Lin M, Chen C, Sun X. End-to-End Zero-Shot HOI Detection via Vision and Language Knowledge Distillation. Proceedings of the AAAI Conference on Artificial Intelligence. 2023;37(3):2839-2846. doi:10.1609/aaai.v37i3.25385
BibTeX
@article{Wu2023,
author = {Wu, Mingrui and Gu, Jiaxin and Shen, Yunhang and Lin, Mingbao and Chen, Chao and Sun, Xiaoshuai},
title = {End-to-End Zero-Shot HOI Detection via Vision and Language Knowledge Distillation},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2023},
volume = {37},
number = {3},
pages = {2839--2846},
doi = {10.1609/aaai.v37i3.25385}
}
EUVIMED