TRACE: Textual Relevance Augmentation and Contextual Encoding for Multimodal Hate Detection
Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (45) · 38763-38771
DOI: 10.1609/aaai.v40i45.41220 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Girish A. Koushik und 3 weiteren Autor:innen.
Abstract (Originalsprache)
Social media memes are a challenging domain for hate detection because they intertwine visual and textual cues into culturally nuanced messages. To tackle these challenges, we introduce TRACE, a hierarchical multimodal framework that leverages visually grounded context augmentation, along with a novel caption-scoring network to emphasize hate-relevant content, and parameter-efficient fine-tuning of CLIP’s text encoder. Our experiments demonstrate that selectively fine-tuning deeper text encoder layers significantly enhances performance compared to simpler projection-layer fine-tuning methods. Specifically, our framework achieves state-of-the-art accuracy (0.807) and F1-score (0.806) on the widely-used Hateful Memes dataset, matching the performance of considerably larger models while maintaining efficiency. Moreover, it achieves superior generalization on the MultiOFF offensive meme dataset (F1-score 0.673), highlighting robustness across meme categories. Additional analyses confirm that robust visual grounding and nuanced text representations significantly reduce errors caused by benign confounders. We publicly release our code to facilitate future research.
Zitieren
APA
Koushik, G. A., Treharne, H., Joshi, A., & Kanojia, D. (2026). TRACE: Textual Relevance Augmentation and Contextual Encoding for Multimodal Hate Detection. Proceedings of the AAAI Conference on Artificial Intelligence, 40(45), 38763-38771. https://doi.org/10.1609/aaai.v40i45.41220
Vancouver
Koushik GA, Treharne H, Joshi A, Kanojia D. TRACE: Textual Relevance Augmentation and Contextual Encoding for Multimodal Hate Detection. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(45):38763-38771. doi:10.1609/aaai.v40i45.41220
BibTeX
@article{Koushik2026,
author = {Koushik, Girish A. and Treharne, Helen and Joshi, Aditya and Kanojia, Diptesh},
title = {TRACE: Textual Relevance Augmentation and Contextual Encoding for Multimodal Hate Detection},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2026},
volume = {40},
number = {45},
pages = {38763--38771},
doi = {10.1609/aaai.v40i45.41220}
}
EUVIMED