DeTAL: Open-Vocabulary Temporal Action Localization With Decoupled Networks
IEEE Transactions on Pattern Analysis and Machine Intelligence · 2024-12 · 46 (12) · 7728-7741
DOI: 10.1109/tpami.2024.3395778 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024-12. Verfasst von Zhiheng Li und 5 weiteren Autor:innen. Die Arbeit wurde bisher 28-mal zitiert (Crossref) und nennt 71 Literaturangaben. Gefördert von: National Natural Science Foundation of China, National Key Research and Development Plan of China, Shandong Excellent Young Scientists Fund Program.
Abstract (Originalsprache)
Pre-trained visual-language (ViL) models have demonstrated good zero-shot capability in video understanding tasks, where they were usually adapted through fine-tuning or temporal modeling. However, in the task of open-vocabulary temporal action localization (OV-TAL), such adaption reduces the robustness of ViL models against different data distributions, leading to a misalignment between visual representations and text descriptions of unseen action categories. As a result, existing methods often strike a trade-off between action detection and classification. Aiming at this issue, this paper proposes DeTAL, a simple but effective two-stage approach for OV-TAL. DeTAL decouples action detection from action classification to avoid the compromise between them, and the state-of-the-art methods for close-set action localization can be handily adapted to OV-TAL, which significantly improves the performance. Meanwhile, DeTAL can easily tackle the scenario where action category annotations are unavailable in the training dataset. In the experiments, we propose a new cross-dataset setting to evaluate the zero-shot capability of different methods. And the results demonstrate that DeTAL outperforms the state-of-the-art methods for OV-TAL on both THUMOS14 and ActivityNet1.3.
Förderung
- National Natural Science Foundation of China
- National Key Research and Development Plan of China
- Shandong Excellent Young Scientists Fund Program
- Self-Developed Innovation Team of Jinan City
- Young Taishan Scholars Program of Shandong Province
Zitieren
APA
Li, Z., Zhong, Y., Song, R., Li, T., Ma, L., & Zhang, W. (2024). DeTAL: Open-Vocabulary Temporal Action Localization With Decoupled Networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(12), 7728-7741. https://doi.org/10.1109/tpami.2024.3395778
Vancouver
Li Z, Zhong Y, Song R, Li T, Ma L, Zhang W. DeTAL: Open-Vocabulary Temporal Action Localization With Decoupled Networks. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2024;46(12):7728-7741. doi:10.1109/tpami.2024.3395778
BibTeX
@article{Li2024,
author = {Li, Zhiheng and Zhong, Yujie and Song, Ran and Li, Tianjiao and Ma, Lin and Zhang, Wei},
title = {DeTAL: Open-Vocabulary Temporal Action Localization With Decoupled Networks},
journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
year = {2024},
volume = {46},
number = {12},
pages = {7728--7741},
doi = {10.1109/tpami.2024.3395778}
}
EUVIMED