Open-Vocabulary Animal Keypoint Detection with Semantic-Feature Matching
International Journal of Computer Vision · 2024-12 · 132 (12) · 5741-5758
DOI: 10.1007/s11263-024-02126-3 ↗Kurz gefasst
Fachartikel in der Zeitschrift International Journal of Computer Vision, online erschienen am 25.06.2024. Verfasst von Hao Zhang und 7 weiteren Autor:innen. Die Arbeit wurde bisher 21-mal zitiert (Crossref) und nennt 56 Literaturangaben.
Abstract (Originalsprache)
Current image-based keypoint detection methods for animal (including human) bodies and faces are generally divided into fully supervised and few-shot class-agnostic approaches. The former typically relies on laborious and time-consuming manual annotations, posing considerable challenges in expanding keypoint detection to a broader range of keypoint categories and animal species. The latter, though less dependent on extensive manual input, still requires necessary support images with annotation for reference during testing. To realize zero-shot keypoint detection without any prior annotation, we introduce the Open-Vocabulary Keypoint Detection (OVKD) task, which is innovatively designed to use text prompts for identifying arbitrary keypoints across any species. In pursuit of this goal, we have developed a novel framework named Open-Vocabulary Keypoint Detection with Semantic-feature Matching (KDSM). This framework synergistically combines vision and language models, creating an interplay between language features and local keypoint visual features. KDSM enhances its capabilities by integrating Domain Distribution Matrix Matching (DDMM) and other special modules, such as the Vision-Keypoint Relational Awareness (VKRA) module, improving the framework’s generalizability and overall performance. Our comprehensive experiments demonstrate that KDSM significantly outperforms the baseline in terms of performance and achieves remarkable success in the OVKD task. Impressively, our method, operating in a zero-shot fashion, still yields results comparable to state-of-the-art few-shot species class-agnostic keypoint detection methods. Codes and data are available at https://github.com/zhanghao5201/KDSM.
Zitieren
APA
Zhang, H., Xu, L., Lai, S., Shao, W., Zheng, N., Luo, P., Qiao, Y., & Zhang, K. (2024). Open-Vocabulary Animal Keypoint Detection with Semantic-Feature Matching. International Journal of Computer Vision, 132(12), 5741-5758. https://doi.org/10.1007/s11263-024-02126-3
Vancouver
Zhang H, Xu L, Lai S, Shao W, Zheng N, Luo P, et al. Open-Vocabulary Animal Keypoint Detection with Semantic-Feature Matching. International Journal of Computer Vision. 2024;132(12):5741-5758. doi:10.1007/s11263-024-02126-3
BibTeX
@article{Zhang2024,
author = {Zhang, Hao and Xu, Lumin and Lai, Shenqi and Shao, Wenqi and Zheng, Nanning and Luo, Ping and Qiao, Yu and Zhang, Kaipeng},
title = {Open-Vocabulary Animal Keypoint Detection with Semantic-Feature Matching},
journal = {International Journal of Computer Vision},
year = {2024},
volume = {132},
number = {12},
pages = {5741--5758},
doi = {10.1007/s11263-024-02126-3}
}
EUVIMED