Multimodal Hateful Meme Classification Based on Transfer Learning and a Cross-Mask Mechanism
Electronics · 15.07.2024 · 13 (14) · 2780
DOI: 10.3390/electronics13142780 ↗Kurz gefasst
Fachartikel in der Zeitschrift Electronics, online erschienen am 15.07.2024. Verfasst von Fan Wu und 4 weiteren Autor:innen. Die Arbeit wurde bisher 14-mal zitiert (Crossref) und nennt 40 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Hainan Province Higher Education Teaching Reform Research Project.
Abstract (Originalsprache)
Hateful memes are malicious and biased sentiment information widely spread on the internet. Detecting hateful memes differs from traditional multimodal tasks because, in conventional tasks, visual and textual information align semantically. However, the challenge in detecting hateful memes lies in their unique multimodal nature, where images and text in memes may be weak or unrelated, requiring models to understand the content and perform multimodal reasoning. To address this issue, we introduce a multimodal fine-grained hateful memes detection model named “TCAM”. The model leverages advanced encoding techniques from TweetEval and CLIP and introduces enhanced Cross-Attention and Cross-Mask Mechanisms (CAM) in the feature fusion stage to improve multimodal correlations. It effectively embeds fine-grained features of data and image descriptions into the model through transfer learning. This paper uses the Area Under the Receiver Operating Characteristic Curve (AUROC) as the primary metric to evaluate the model’s discriminatory ability. This approach achieved an AUROC score of 0.8362 and an accuracy score of 0.764 on the Facebook Hateful Memes Challenge (FHMC) dataset, confirming its high discriminatory capability. The TCAM model demonstrates relatively superior performance compared to ensemble machine learning methods.
Förderung
- Hainan Province Higher Education Teaching Reform Research Project
Zitieren
APA
Wu, F., Chen, G., Cao, J., Yan, Y., & Li, Z. (2024). Multimodal Hateful Meme Classification Based on Transfer Learning and a Cross-Mask Mechanism. Electronics, 13(14), 2780. https://doi.org/10.3390/electronics13142780
Vancouver
Wu F, Chen G, Cao J, Yan Y, Li Z. Multimodal Hateful Meme Classification Based on Transfer Learning and a Cross-Mask Mechanism. Electronics. 2024;13(14):2780. doi:10.3390/electronics13142780
BibTeX
@article{Wu2024,
author = {Wu, Fan and Chen, Guolian and Cao, Junkuo and Yan, Yuhan and Li, Zhongneng},
title = {Multimodal Hateful Meme Classification Based on Transfer Learning and a Cross-Mask Mechanism},
journal = {Electronics},
year = {2024},
volume = {13},
number = {14},
pages = {2780},
doi = {10.3390/electronics13142780}
}
EUVIMED