EUVIMEDAtrasti EUVIMED
Crossref

${\text{CA}^{2}\text{ST}}$: Cross-Attention in Audio, Space, and Time for Holistic Video Recognition

Jongseo Lee, Joohyun Chang, Dongho Lee, Jinwoo Choi

IEEE Transactions on Pattern Analysis and Machine Intelligence · 2026-03 · 48 (3) · 2803-2819

DOI: 10.1109/tpami.2025.3628653 ↗

At a glance

Journal article in IEEE Transactions on Pattern Analysis and Machine Intelligence, 2026-03. Written by Jongseo Lee and 3 co-authors. Cited 5 times so far (Crossref); 105 references. Funded by: Kyung Hee University in 2024.

SantraukaBibliografiniai duomenysCiteRelated studies

Abstract (original language)

We propose Cross-Attention in Audio, Space, and Time (C$\text{A}^{2}$A2ST), a transformer-based method for holistic video recognition. Recognizing actions in videos requires both spatial and temporal understanding, yet most existing models lack a balanced spatio-temporal understanding of videos. To address this, we propose a novel two-stream architecture, called Cross-Attention in Space and Time (CAST), using only RGB input. In each layer of CAST, Bottleneck Cross-Attention (B-CA) enables spatial and temporal experts to exchange information and make synergistic predictions. For holistic video understanding, we extend CAST by integrating an audio expert, forming Cross-Attention in Visual and Audio (CAVA). We validate the CAST on benchmarks with different characteristics, EPIC-KITCHENS-100, Something-Something-V2, Kinetics-400, ActivityNet, and HD-EPIC to show balanced performance. We also validate the CAVA on audio-visual action recognition benchmarks, including UCF-101, VGG-Sound, KineticsSound, EPIC- SOUNDS, and HD-EPIC-SOUNDS. CAVA shows favorable performance on these datasets, demonstrating the effective information exchange among multiple experts within the B-CA module. In addition, C$\text{A}^{2}$A2ST combines CAST and CAVA by employing spatial, temporal, and audio experts through cross-attention, achieving balanced and holistic video understanding.

Funding

  • Kyung Hee University in 2024

Cite

APA

Lee, J., Chang, J., Lee, D., & Choi, J. (2026). ${\text{CA}^{2}\text{ST}}$: Cross-Attention in Audio, Space, and Time for Holistic Video Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 48(3), 2803-2819. https://doi.org/10.1109/tpami.2025.3628653

Vancouver

Lee J, Chang J, Lee D, Choi J. ${\text{CA}^{2}\text{ST}}$: Cross-Attention in Audio, Space, and Time for Holistic Video Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2026;48(3):2803-2819. doi:10.1109/tpami.2025.3628653

BibTeX
@article{Lee2026,
  author = {Lee, Jongseo and Chang, Joohyun and Lee, Dongho and Choi, Jinwoo},
  title = {$\textCA^2\textST$: Cross-Attention in Audio, Space, and Time for Holistic Video Recognition},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2026},
  volume = {48},
  number = {3},
  pages = {2803--2819},
  doi = {10.1109/tpami.2025.3628653}
}
Start for free →