Voice-Face Homogeneity Tells Deepfake
ACM Transactions on Multimedia Computing, Communications, and Applications · 31.03.2024 · 20 (3) · 1-22
DOI: 10.1145/3625231 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Multimedia Computing, Communications, and Applications, online erschienen am 10.11.2023. Verfasst von Harry Cheng und 5 weiteren Autor:innen. Die Arbeit wurde bisher 97-mal zitiert (Crossref) und nennt 74 Literaturangaben.
Abstract (Originalsprache)
Detecting forgery videos is highly desirable due to the abuse of deepfake. Existing detection approaches contribute to exploring the specific artifacts in deepfake videos and fit well on certain data. However, the growing technique on these artifacts keeps challenging the robustness of traditional deepfake detectors. As a result, the development of these approaches has reached a blockage. In this article, we propose to perform deepfake detection from an unexplored voice-face matching view. Our approach is founded on two supporting points: first, there is a high degree of homogeneity between the voice and face of an individual (i.e., they are highly correlated), and second, deepfake videos often involve mismatched identities between the voice and face due to face-swapping techniques. To this end, we develop a voice-face matching method that measures the matching degree between these two modalities to identify deepfake videos. Nevertheless, training on specific deepfake datasets makes the model overfit certain traits of deepfake algorithms. We instead advocate a method that quickly adapts to untapped forgery, with a pre-training then fine-tuning paradigm. Specifically, we first pre-train the model on a generic audio-visual dataset, followed by the fine-tuning on downstream deepfake data. We conduct extensive experiments over three widely exploited deepfake datasets: DFDC, FakeAVCeleb, and DeepfakeTIMIT. Our method obtains significant performance gains as compared to other state-of-the-art competitors. For instance, our method outperforms the baselines by nearly 2%, achieving an AUC of 86.11% on FakeAVCeleb. It is also worth noting that our method already achieves competitive results when fine-tuned on limited deepfake data.
Zitieren
APA
Cheng, H., Guo, Y., Wang, T., Li, Q., Chang, X., & Nie, L. (2024). Voice-Face Homogeneity Tells Deepfake. ACM Transactions on Multimedia Computing, Communications, and Applications, 20(3), 1-22. https://doi.org/10.1145/3625231
Vancouver
Cheng H, Guo Y, Wang T, Li Q, Chang X, Nie L. Voice-Face Homogeneity Tells Deepfake. ACM Transactions on Multimedia Computing, Communications, and Applications. 2024;20(3):1-22. doi:10.1145/3625231
BibTeX
@article{Cheng2024,
author = {Cheng, Harry and Guo, Yangyang and Wang, Tianyi and Li, Qi and Chang, Xiaojun and Nie, Liqiang},
title = {Voice-Face Homogeneity Tells Deepfake},
journal = {ACM Transactions on Multimedia Computing, Communications, and Applications},
year = {2024},
volume = {20},
number = {3},
pages = {1--22},
doi = {10.1145/3625231}
}
EUVIMED