Emotions Don't Lie: An Audio-Visual Deepfake Detection Method Using Affective Cues
We present a learning-based method for detecting real and fake deepfake multimedia content. To maximize information for learning, we extract and analyze the similarity between the two audio and visual modalities from within the same video. Additionally, we extract and compare affective cues corresponding to perceived emotion from the two modalities within a video to infer whether the input video is "real" or "fake". We propose a deep learning network, inspired by the Siamese network architecture and the triplet loss. To validate our model, we report the AUC metric on two large-scale deepfake detection datasets, DeepFake-TIMIT Dataset and DFDC. We compare our approach with several SOTA deepfake detection methods and report per-video AUC of 84.4% on the DFDC and 96.6% on the DF-TIMIT datasets, respectively. To the best of our knowledge, ours is the first approach that simultaneously exploits audio and video modalities and also perceived emotions from the two modalities for deepfake detection.
Code (0)
등록된 구현이 없습니다.
Tasks
DeepFake DetectionFace SwappingTripletMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
EMO-BOOST: Emotion-Augmented Audio-Visual Features for Improved Generalization in Deepfake Detection
With every advancement in generative AI models, forensics is under increasing pressure. The constant emergence of new generation techniques makes it impossible to collect data for each manipulation to train a deepfake de…
Emotion RecognitionDeepFake DetectionMIS-AVoiDD: Modality Invariant and Specific Representation for Audio-Visual Deepfake Detection
Deepfakes are synthetic media generated using deep generative algorithms and have posed a severe societal and political threat. Apart from facial manipulation and synthetic voice, recently, a novel kind of deepfakes has …
DeepFake DetectionFace SwappingJoint Audio-Visual Attention with Contrastive Learning for More General Deepfake Detection
With the continuous advancement of deepfake technology, there has been a surge in the creation of realistic fake videos. Unfortunately, the malicious utilization of deepfake poses a significant threat to societal moralit…
Contrastive LearningDeepFake DetectionFace SwappingHuman Detection of DeepfakesContextual Cross-Modal Attention for Audio-Visual Deepfake Detection and Localization
In the digital age, the emergence of deepfakes and synthetic media presents a significant threat to societal and political integrity. Deepfakes based on multi-modal manipulation, such as audio-visual, are more realistic …
DeepFake DetectionFace SwappingIntegrating Audio-Visual Features for Multimodal Deepfake Detection
Deepfakes are AI-generated media in which an image or video has been digitally modified. The advancements made in deepfake technology have led to privacy and security issues. Most deepfake detection techniques rely on th…
Binary ClassificationDeepFake DetectionFace Swapping