A Multimodal Hierarchical Attentional Ordering Network
ACM Transactions on Multimedia Computing, Communications, and Applications · 31.03.2025 · 21 (3) · 1-20
DOI: 10.1145/3711864 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Multimedia Computing, Communications, and Applications, online erschienen am 19.02.2025. Verfasst von Yiping Yang und 2 weiteren Autor:innen. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 42 Literaturangaben. Gefördert von: National Key R&D Program of China, NSFC, Key R&D Program of Zhejiang Province.
Abstract (Originalsprache)
Sequence coherence modeling refers to how to coherently organize a given set of elements within a sequence, which is a fundamental aspect in comprehension, generation, and reasoning tasks. Although making great progress, existing deep neural network-based ordering approaches suffer from critical limitations: (1) Most of the previous coherence modeling approaches focus on the ordering of pure text sequences, while real-world data are often expressed in multimodal form; (2) The pairwise ranking method, widely used for ordering single/multimodal sequences, faces challenges in effectively modeling overall contextual relationships; (3) The redundancy problem would exist in multimodal ordering and can influence the encoding performance, which has been ignored when performing ordering. In this article, we propose a novel Multimodal Hierarchical Attentional Ordering Network (referred to as MHAONet) for multimodal sequence ordering, which effectively captures the required global dependencies in multimodal sequences to enhance the coherence modeling. In particular, a multimodal encoder is introduced to encode the whole sequence for fully exploiting global contexts, where a multimodal selectively visible mask module is elaborately designed for controlled information transmission between different modalities. This enables the model to capture multimodal semantic and contextual information across the sequence. Further, the token-level and paragraph-level hierarchical attention layers are employed to encourage the model to concentrate on features of different granularities. In addition, a transformer-based pointer network is used for performing order prediction. Comprehensive experiments and analysis on benchmark datasets demonstrate the effectiveness and superiority of our proposed model.
Förderung
- National Key R&D Program of China
- NSFC
- Key R&D Program of Zhejiang Province
Zitieren
APA
Yang, Y., Cui, B., & Li, Y. (2025). A Multimodal Hierarchical Attentional Ordering Network. ACM Transactions on Multimedia Computing, Communications, and Applications, 21(3), 1-20. https://doi.org/10.1145/3711864
Vancouver
Yang Y, Cui B, Li Y. A Multimodal Hierarchical Attentional Ordering Network. ACM Transactions on Multimedia Computing, Communications, and Applications. 2025;21(3):1-20. doi:10.1145/3711864
BibTeX
@article{Yang2025,
author = {Yang, Yiping and Cui, Baiyun and Li, Yingming},
title = {A Multimodal Hierarchical Attentional Ordering Network},
journal = {ACM Transactions on Multimedia Computing, Communications, and Applications},
year = {2025},
volume = {21},
number = {3},
pages = {1--20},
doi = {10.1145/3711864}
}
EUVIMED