Multi-Granular Multimodal Clue Fusion for Meme Understanding
Proceedings of the AAAI Conference on Artificial Intelligence · 11.04.2025 · 39 (24) · 26057-26065
DOI: 10.1609/aaai.v39i24.34801 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 11.04.2025. Verfasst von Li Zheng und 7 weiteren Autor:innen. Die Arbeit wurde bisher 5-mal zitiert (Crossref).
Abstract (Originalsprache)
With the continuous emergence of various social media platforms frequently used in daily life, the multimodal meme understanding (MMU) task has been garnering increasing attention. MMU aims to explore and comprehend the meanings of memes from various perspectives by performing tasks such as metaphor recognition, sentiment analysis, intention detection, and offensiveness detection. Despite making progress, limitations persist due to the loss of fine-grained metaphorical visual clue and the neglect of multimodal text-image weak correlation. To overcome these limitations, we propose a multi-granular multimodal clue fusion model (MGMCF) to advance MMU. Firstly, we design an object-level semantic mining module to extract object-level image feature clues, achieving fine-grained feature clue extraction and enhancing the model's ability to capture metaphorical details and semantics. Secondly, we propose a brand-new global-local cross-modal interaction model to address the weak correlation between text and images. This model facilitates effective interaction between global multimodal contextual clues and local unimodal feature clues, strengthening their representations through a bidirectional cross-modal attention mechanism. Finally, we devise a dual-semantic guided training strategy to enhance the model's understanding and alignment of multimodal representations in the semantic space. Experiments conducted on the widely-used MET-MEME bilingual dataset demonstrate significant improvements over state-of-the-art baselines. Specifically, there is an 8.14% increase in precision for offensiveness detection task, and respective accuracy enhancements of 3.53%, 3.89%, and 3.52% for metaphor recognition, sentiment analysis, and intention detection tasks. These results, underpinned by in-depth analyses, underscore the effectiveness and potential of our approach for advancing MMU.
Zitieren
APA
Zheng, L., Fei, H., Dai, T., Peng, Z., Li, F., Ma, H., Teng, C., & Ji, D. (2025). Multi-Granular Multimodal Clue Fusion for Meme Understanding. Proceedings of the AAAI Conference on Artificial Intelligence, 39(24), 26057-26065. https://doi.org/10.1609/aaai.v39i24.34801
Vancouver
Zheng L, Fei H, Dai T, Peng Z, Li F, Ma H, et al. Multi-Granular Multimodal Clue Fusion for Meme Understanding. Proceedings of the AAAI Conference on Artificial Intelligence. 2025;39(24):26057-26065. doi:10.1609/aaai.v39i24.34801
BibTeX
@article{Zheng2025,
author = {Zheng, Li and Fei, Hao and Dai, Ting and Peng, Zuquan and Li, Fei and Ma, Huisheng and Teng, Chong and Ji, Donghong},
title = {Multi-Granular Multimodal Clue Fusion for Meme Understanding},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2025},
volume = {39},
number = {24},
pages = {26057--26065},
doi = {10.1609/aaai.v39i24.34801}
}
EUVIMED