EUVIMEDEUVIMED entdecken
Crossref

TRACE: Textual Relevance Augmentation and Contextual Encoding for Multimodal Hate Detection

Girish A. Koushik, Helen Treharne, Aditya Joshi, Diptesh Kanojia

Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (45) · 38763-38771

DOI: 10.1609/aaai.v40i45.41220 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Girish A. Koushik und 3 weiteren Autor:innen.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Social media memes are a challenging domain for hate detection because they intertwine visual and textual cues into culturally nuanced messages. To tackle these challenges, we introduce TRACE, a hierarchical multimodal framework that leverages visually grounded context augmentation, along with a novel caption-scoring network to emphasize hate-relevant content, and parameter-efficient fine-tuning of CLIP’s text encoder. Our experiments demonstrate that selectively fine-tuning deeper text encoder layers significantly enhances performance compared to simpler projection-layer fine-tuning methods. Specifically, our framework achieves state-of-the-art accuracy (0.807) and F1-score (0.806) on the widely-used Hateful Memes dataset, matching the performance of considerably larger models while maintaining efficiency. Moreover, it achieves superior generalization on the MultiOFF offensive meme dataset (F1-score 0.673), highlighting robustness across meme categories. Additional analyses confirm that robust visual grounding and nuanced text representations significantly reduce errors caused by benign confounders. We publicly release our code to facilitate future research.

Zitieren

APA

Koushik, G. A., Treharne, H., Joshi, A., & Kanojia, D. (2026). TRACE: Textual Relevance Augmentation and Contextual Encoding for Multimodal Hate Detection. Proceedings of the AAAI Conference on Artificial Intelligence, 40(45), 38763-38771. https://doi.org/10.1609/aaai.v40i45.41220

Vancouver

Koushik GA, Treharne H, Joshi A, Kanojia D. TRACE: Textual Relevance Augmentation and Contextual Encoding for Multimodal Hate Detection. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(45):38763-38771. doi:10.1609/aaai.v40i45.41220

BibTeX
@article{Koushik2026,
  author = {Koushik, Girish A. and Treharne, Helen and Joshi, Aditya and Kanojia, Diptesh},
  title = {TRACE: Textual Relevance Augmentation and Contextual Encoding for Multimodal Hate Detection},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2026},
  volume = {40},
  number = {45},
  pages = {38763--38771},
  doi = {10.1609/aaai.v40i45.41220}
}
Kostenlos starten →