COSMOS: Catching Out-of-Context Image Misuse Using Self-Supervised Learning
Proceedings of the AAAI Conference on Artificial Intelligence · 26.06.2023 · 37 (12) · 14084-14092
DOI: 10.1609/aaai.v37i12.26648 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 26.06.2023. Verfasst von Shivangi Aneja und 2 weiteren Autor:innen. Die Arbeit wurde bisher 29-mal zitiert (Crossref).
Abstract (Originalsprache)
Despite the recent attention to DeepFakes, one of the most prevalent ways to mislead audiences on social media is the use of unaltered images in a new but false context. We propose a new method that automatically highlights out-of-context image and text pairs, for assisting fact-checkers. Our key insight is to leverage the grounding of images with text to distinguish out-of-context scenarios that cannot be disambiguated with language alone. We propose a self-supervised training strategy where we only need a set of captioned images. At train time, our method learns to selectively align individual objects in an image with textual claims, without explicit supervision. At test time, we check if both captions correspond to the same object(s) in the image but are semantically different, which allows us to make fairly accurate out-of-context predictions. Our method achieves 85% out-of-context detection accuracy. To facilitate benchmarking of this task, we create a large-scale dataset of 200K images with 450K textual captions from a variety of news websites, blogs, and social media posts
Zitieren
APA
Aneja, S., Bregler, C., & Niessner, M. (2023). COSMOS: Catching Out-of-Context Image Misuse Using Self-Supervised Learning. Proceedings of the AAAI Conference on Artificial Intelligence, 37(12), 14084-14092. https://doi.org/10.1609/aaai.v37i12.26648
Vancouver
Aneja S, Bregler C, Niessner M. COSMOS: Catching Out-of-Context Image Misuse Using Self-Supervised Learning. Proceedings of the AAAI Conference on Artificial Intelligence. 2023;37(12):14084-14092. doi:10.1609/aaai.v37i12.26648
BibTeX
@article{Aneja2023,
author = {Aneja, Shivangi and Bregler, Chris and Niessner, Matthias},
title = {COSMOS: Catching Out-of-Context Image Misuse Using Self-Supervised Learning},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2023},
volume = {37},
number = {12},
pages = {14084--14092},
doi = {10.1609/aaai.v37i12.26648}
}
EUVIMED