EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

A Robust Approach to Multimodal Deepfake Detection

Davide Salvi iD, Honggu Liu iD, Sara Mandelli, Paolo Bestagini, Wenbo Zhou, Weiming Zhang, Stefano Tubaro

Journal of Imaging · 19.06.2023 · 9 (6) · 122

DOI: 10.3390/jimaging9060122 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Imaging, online erschienen am 19.06.2023. Verfasst von Davide Salvi und 6 weiteren Autor:innen. Beteiligte Einrichtung: Dipartimento di Elettronica, Informazione e Bioingegneria, Politecnico di Milano, 20133 Milan, Italy. Die Arbeit wurde bisher 100-mal zitiert (Crossref) und nennt 57 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Defense Advanced Research Projects Agency, European Union under the Italian National Recovery and Resilience Plan (NRRP), European Union.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The widespread use of deep learning techniques for creating realistic synthetic media, commonly known as deepfakes, poses a significant threat to individuals, organizations, and society. As the malicious use of these data could lead to unpleasant situations, it is becoming crucial to distinguish between authentic and fake media. Nonetheless, though deepfake generation systems can create convincing images and audio, they may struggle to maintain consistency across different data modalities, such as producing a realistic video sequence where both visual frames and speech are fake and consistent one with the other. Moreover, these systems may not accurately reproduce semantic and timely accurate aspects. All these elements can be exploited to perform a robust detection of fake content. In this paper, we propose a novel approach for detecting deepfake video sequences by leveraging data multimodality. Our method extracts audio-visual features from the input video over time and analyzes them using time-aware neural networks. We exploit both the video and audio modalities to leverage the inconsistencies between and within them, enhancing the final detection performance. The peculiarity of the proposed method is that we never train on multimodal deepfake data, but on disjoint monomodal datasets which contain visual-only or audio-only deepfakes. This frees us from leveraging multimodal datasets during training, which is desirable given their lack in the literature. Moreover, at test time, it allows to evaluate the robustness of our proposed detector on unseen multimodal deepfakes. We test different fusion techniques between data modalities and investigate which one leads to more robust predictions by the developed detectors. Our results indicate that a multimodal approach is more effective than a monomodal one, even if trained on disjoint monomodal datasets.

Schlagwörter

Stichwörter der Autor:innen

audio forensicsdeepfake detectionmultimodalityvideo forensics

Autor:innen und Institutionen

  • Dipartimento di Elettronica, Informazione e Bioingegneria, Politecnico di Milano, 20133 Milan, Italy.
  • School of Cyber Science and Technology, University of Science and Technology of China, Hefei 230026, China.

ORCID-Profile: Davide Salvi, Honggu Liu

Förderung

  • Defense Advanced Research Projects Agency
  • European Union under the Italian National Recovery and Resilience Plan (NRRP)
  • European Union

Zitieren

APA

Salvi, D., Liu, H., Mandelli, S., Bestagini, P., Zhou, W., Zhang, W., & Tubaro, S. (2023). A Robust Approach to Multimodal Deepfake Detection. Journal of Imaging, 9(6), 122. https://doi.org/10.3390/jimaging9060122

Vancouver

Salvi D, Liu H, Mandelli S, Bestagini P, Zhou W, Zhang W, et al. A Robust Approach to Multimodal Deepfake Detection. Journal of Imaging. 2023;9(6):122. doi:10.3390/jimaging9060122

BibTeX
@article{Salvi2023,
  author = {Salvi, Davide and Liu, Honggu and Mandelli, Sara and Bestagini, Paolo and Zhou, Wenbo and Zhang, Weiming and Tubaro, Stefano},
  title = {A Robust Approach to Multimodal Deepfake Detection},
  journal = {Journal of Imaging},
  year = {2023},
  volume = {9},
  number = {6},
  pages = {122},
  doi = {10.3390/jimaging9060122}
}
Kostenlos starten →