Exploring Self-Supervised Vision Transformers for Deepfake Detection: A Comparative Analysis
This paper investigates the effectiveness of self-supervised pre-trained vision transformers (ViTs) compared to supervised pre-trained ViTs and conventional neural networks (ConvNets) for detecting facial deepfake images and videos. It examines their potential for improved generalization and explainability, especially with limited training data. Despite the success of transformer architectures in various tasks, the deepfake detection community is hesitant to use large ViTs as feature extractors due to their perceived need for extensive data and suboptimal generalization with small datasets. This contrasts with ConvNets, which are already established as robust feature extractors. Additionally, training ViTs from scratch requires significant resources, limiting their use to large companies. Recent advancements in self-supervised learning (SSL) for ViTs, like masked autoencoders and DINOs, show adaptability across diverse tasks and semantic segmentation capabilities. By leveraging SSL ViTs for deepfake detection with modest data and partial fine-tuning, we find comparable adaptability to deepfake detection and explainability via the attention mechanism. Moreover, partial fine-tuning of ViTs is a resource-efficient option.
Code (1)
Tasks
DeepFake DetectionFace SwappingFew-Shot LearningSelf-Supervised LearningSemantic SegmentationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Deepfake Detection: A Comparative Analysis
This paper present a comprehensive comparative analysis of supervised and self-supervised models for deepfake detection. We evaluate eight supervised deep learning architectures and two transformer-based models pre-train…
DeepFake DetectionDeep LearningFace SwappingHow Close are Other Computer Vision Tasks to Deepfake Detection?
In this paper, we challenge the conventional belief that supervised ImageNet-trained models have strong generalizability and are suitable for use as feature extractors in deepfake detection. We present a new measurement,…
DeepFake DetectionFace RecognitionFace SwappingExploring Adversarial Attacks and Defenses in Vision Transformers trained with DINO
This work conducts the first analysis on the robustness against adversarial attacks on self-supervised Vision Transformers trained using DINO. First, we evaluate whether features learned through self-supervision are more…
Adversarial RobustnessUNTAG: LEARNING GENERIC FEATURES FOR UNSUPERVISED TYPE-AGNOSTIC DEEPFAKE DETECTION
This paper introduces a novel framework for unsupervised type-agnostic deepfake detection called UNTAG. Existing methods are generally trained in a supervised manner at the classification level, focusing on detecting at …
DeepFake DetectionFace SwappingOne-Class ClassificationOne-class classifierSERE: Exploring Feature Self-relation for Self-supervised Transformer
Learning representations with self-supervision for convolutional networks (CNN) has been validated to be effective for vision tasks. As an alternative to CNN, vision transformers (ViT) have strong representation ability …
RelationSelf-Supervised LearningSemantic SegmentationUnsupervised Semantic Segmentation