EUVIMEDEUVIMED entdecken
Crossref

Detecting and Grounding Multi-Modal Media Manipulation and Beyond

Rui Shao, Tianxing Wu, Jianlong Wu, Liqiang Nie, Ziwei Liu

IEEE Transactions on Pattern Analysis and Machine Intelligence · 2024-08 · 46 (8) · 5556-5574

DOI: 10.1109/tpami.2024.3367749 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024-08. Verfasst von Rui Shao und 4 weiteren Autor:innen. Die Arbeit wurde bisher 69-mal zitiert (Crossref) und nennt 72 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Young Elite Scientists Sponsorship Program by CAST, Ministry of Education - Singapore.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Misinformation has become a pressing issue. Fake media, in both visual and textual forms, is widespread on the web. While various DeepFake detection and text fake news detection methods have been proposed, they are only designed for single-modality forgery based on binary classification, let alone analyzing and reasoning subtle forgery traces across different modalities. In this paper, we highlight a new research problem for multi-modal fake media, namely Detecting and Grounding Multi-Modal Media Manipulation (DGM 4). DGM 4 aims to not only detect the authenticity of multi-modal media, but also ground the manipulated content (i.e., image bounding boxes and text tokens), which requires deeper reasoning of multi-modal media manipulation. To support a large-scale investigation, we construct the first DGM 4 dataset, where image-text pairs are manipulated by various approaches, with rich annotation of diverse manipulations. Moreover, we propose a novel HierArchical Multi-modal Manipulation rEasoning tRansformer (HAMMER) to fully capture the fine-grained interaction between different modalities. HAMMER performs: 1) manipulation-aware contrastive learning between two uni-modal encoders as shallow manipulation reasoning and 2) modality-aware cross-attention by multi-modal aggregator as deep manipulation reasoning. Dedicated manipulation detection and grounding heads are integrated from shallow to deep levels based on the interacted multi-modal information. To exploit more fine-grained contrastive learning for cross-modal semantic alignment, we further integrate Manipulation-Aware Contrastive Loss with Local View and construct a more advanced model HAMMER++. Finally, we build an extensive benchmark and set up rigorous evaluation metrics for this new research problem. Comprehensive experiments demonstrate the superiority of HAMMER and HAMMER++; several valuable observations are also revealed to facilitate future research in multi-modal media manipulation.

Förderung

  • National Natural Science Foundation of China
  • Young Elite Scientists Sponsorship Program by CAST
  • Ministry of Education - Singapore
  • MOE AcRF Tier 2
  • NTU NAP
  • RIE2020 Industry Alignment Fund

Zitieren

APA

Shao, R., Wu, T., Wu, J., Nie, L., & Liu, Z. (2024). Detecting and Grounding Multi-Modal Media Manipulation and Beyond. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(8), 5556-5574. https://doi.org/10.1109/tpami.2024.3367749

Vancouver

Shao R, Wu T, Wu J, Nie L, Liu Z. Detecting and Grounding Multi-Modal Media Manipulation and Beyond. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2024;46(8):5556-5574. doi:10.1109/tpami.2024.3367749

BibTeX
@article{Shao2024,
  author = {Shao, Rui and Wu, Tianxing and Wu, Jianlong and Nie, Liqiang and Liu, Ziwei},
  title = {Detecting and Grounding Multi-Modal Media Manipulation and Beyond},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2024},
  volume = {46},
  number = {8},
  pages = {5556--5574},
  doi = {10.1109/tpami.2024.3367749}
}
Kostenlos starten →