Spatio-Temporal Disentanglement and Constrained Self-Attention for Multi-Modal Deception Detection
ACM Transactions on Multimedia Computing, Communications, and Applications · 28.02.2026 · 22 (2) · 1-20
DOI: 10.1145/3783994 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Multimedia Computing, Communications, and Applications, online erschienen am 09.02.2026. Verfasst von Zhiwen Shao und 6 weiteren Autor:innen. Die Arbeit wurde bisher 25-mal zitiert (Crossref) und nennt 45 Literaturangaben. Gefördert von: Science and Technology Program of Xuzhou, National Natural Science Foundation of China, Opening Fund of the State Key Laboratory of Industrial Control Technology, China.
Abstract (Originalsprache)
Multi-modal deception detection is a challenging yet important task, having pivotal applications in many fields such as business credibility assessment and multimedia anti-frauds. Previous methods either rely solely on spatial features or overemphasize only temporal information within or across modalities, which may overlook potential critical clues. Motivated by these observations, we propose a Spatio-Temporal Representation Disentanglement (STRD) framework for multi-modal deception detection, which uses a dual-encoder structure to learn spatial and temporal representations for each modality. Specifically, we introduce a pre-trained foundation model to act as the spatial encoder and design a lightweight network as the temporal encoder, extracting spatial semantics and capturing dynamic temporal patterns. Then, we propose a Constrained Self-Attention Block (CSAB), in which self-attention distribution of each head is regarded as spatial distribution and is constrained to attend a certain facial local region. Furthermore, we present a Cross-Modal Correlation Fusion Block (CCFB) to achieve temporal synchronization across modalities by measuring the correlations between visual and audio features. Extensive experiments show that our STRD outperforms the state-of-the-art methods on challenging DOLOS, BOL, BgOL, and RLtrial benchmarks. Particularly, STRD improves by 2.12% and 1.88% over the previous best results in terms of ACC on the DOLOS and BOL datasets, respectively. Additionally, STRD outperforms previous methods in cross-dataset testing, highlighting its superior generalization ability.
Förderung
- Science and Technology Program of Xuzhou
- National Natural Science Foundation of China
- Opening Fund of the State Key Laboratory of Industrial Control Technology, China
- Natural Science Foundation of Shanghai
Zitieren
APA
Shao, Z., Yang, H., Zhu, H., Yao, R., Zou, L., Li, M., & Sheng, B. (2026). Spatio-Temporal Disentanglement and Constrained Self-Attention for Multi-Modal Deception Detection. ACM Transactions on Multimedia Computing, Communications, and Applications, 22(2), 1-20. https://doi.org/10.1145/3783994
Vancouver
Shao Z, Yang H, Zhu H, Yao R, Zou L, Li M, et al. Spatio-Temporal Disentanglement and Constrained Self-Attention for Multi-Modal Deception Detection. ACM Transactions on Multimedia Computing, Communications, and Applications. 2026;22(2):1-20. doi:10.1145/3783994
BibTeX
@article{Shao2026,
author = {Shao, Zhiwen and Yang, Hang and Zhu, Hancheng and Yao, Rui and Zou, Lixin and Li, Mengtian and Sheng, Bin},
title = {Spatio-Temporal Disentanglement and Constrained Self-Attention for Multi-Modal Deception Detection},
journal = {ACM Transactions on Multimedia Computing, Communications, and Applications},
year = {2026},
volume = {22},
number = {2},
pages = {1--20},
doi = {10.1145/3783994}
}
EUVIMED