EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

SASA-CLIP: Structure-Aware Alignment with a Gaussian Prior for Fine-Grained Video Action Recognition

Xiaowei Han, Wenbao Si, Honghui Zhang, Maolin Yang, Lin Ma, Yibo Feng

Sensors · 02.08.2026 · 26 (15) · 4865

DOI: 10.3390/s26154865 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Sensors, online erschienen am 02.08.2026. Verfasst von Xiaowei Han und 5 weiteren Autor:innen. Beteiligte Einrichtung: School of Computer and Information Engineering, Harbin University of Commerce, Harbin 150028, China. Die Arbeit nennt 34 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Heilongjiang Provincial Natural Science Foundation of China.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Fine-grained video action recognition remains challenging because action categories often differ only in subtle inter-class variations and complex temporal dynamics. Recent Contrastive Language–Image Pre-training (CLIP)-based extensions perform well on general action recognition, but they typically rely on early global pooling of video features. Such coarse representations discard the fine temporal cues that distinguish subtle actions, causing a granularity mismatch in cross-modal alignment. To address this, we propose Structure-Aware Semantic-Adaptive (SASA)-CLIP, a framework for multi-granular cross-modal alignment. SASA-CLIP adopts a dual-branch design: a coarse-grained branch captures the global context, while a fine-grained branch matches descriptors against individual frames before aggregation, rather than pooling features early. To keep this alignment temporally coherent, we introduce a Gaussian prior as a temporal structural constraint, encoding the inductive bias of local temporal continuity into the attention matrix to guide an ordered alignment of key action segments along the temporal axis. On Kinetics-400 (ViT-B/32), SASA-CLIP reaches a Top-1 accuracy of 81.37%, improving over the X-CLIP baseline by 0.97%; on HMDB-51 and UCF-101 (ViT-B/16), it reaches 74.0% and 96.81%, improving by 3.25% and 2.61%, respectively. It also transfers to the zero-shot setting, improving over the baseline on HMDB-51 and UCF-101. These results show that combining multi-granular representations with a temporal structural prior benefits fine-grained recognition, suggesting that SASA-CLIP is a practical option for real-world visual sensing applications such as intelligent surveillance and wearable activity monitoring.

Schlagwörter

Stichwörter der Autor:innen

CLIPcross-modal alignmentdeep visual recognitionfine-grained action recognitionvideo understandingvision–language models

Autor:innen und Institutionen

  • School of Computer and Information Engineering, Harbin University of Commerce, Harbin 150028, China.
  • Heilongjiang Provincial Key Laboratory of Electronic Commerce and Information Processing, Harbin 150028, China.

Förderung

  • Heilongjiang Provincial Natural Science Foundation of China

Zitieren

APA

Han, X., Si, W., Zhang, H., Yang, M., Ma, L., & Feng, Y. (2026). SASA-CLIP: Structure-Aware Alignment with a Gaussian Prior for Fine-Grained Video Action Recognition. Sensors, 26(15), 4865. https://doi.org/10.3390/s26154865

Vancouver

Han X, Si W, Zhang H, Yang M, Ma L, Feng Y. SASA-CLIP: Structure-Aware Alignment with a Gaussian Prior for Fine-Grained Video Action Recognition. Sensors. 2026;26(15):4865. doi:10.3390/s26154865

BibTeX
@article{Han2026,
  author = {Han, Xiaowei and Si, Wenbao and Zhang, Honghui and Yang, Maolin and Ma, Lin and Feng, Yibo},
  title = {SASA-CLIP: Structure-Aware Alignment with a Gaussian Prior for Fine-Grained Video Action Recognition},
  journal = {Sensors},
  year = {2026},
  volume = {26},
  number = {15},
  pages = {4865},
  doi = {10.3390/s26154865}
}
Kostenlos starten →