Is Self-Supervised Learning More Robust Than Supervised Learning?
Self-supervised contrastive learning is a powerful tool to learn visual representation without labels. Prior work has primarily focused on evaluating the recognition accuracy of various pre-training algorithms, but has overlooked other behavioral aspects. In addition to accuracy, distributional robustness plays a critical role in the reliability of machine learning models. We design and conduct a series of robustness tests to quantify the behavioral differences between contrastive learning and supervised learning to downstream or pre-training data distribution changes. These tests leverage data corruptions at multiple levels, ranging from pixel-level gamma distortion to patch-level shuffling and to dataset-level distribution shift. Our tests unveil intriguing robustness behaviors of contrastive and supervised learning. On the one hand, under downstream corruptions, we generally observe that contrastive learning is surprisingly more robust than supervised learning. On the other hand, under pre-training corruptions, we find contrastive learning vulnerable to patch shuffling and pixel intensity change, yet less sensitive to dataset-level distribution change. We attempt to explain these results through the role of data augmentation and feature space properties. Our insight has implications in improving the downstream robustness of supervised learning.
Code (0)
등록된 구현이 없습니다.
Tasks
Contrastive LearningData AugmentationSelf-Supervised LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
A Study on Self-Supervised Pretraining for Vision Problems in Gastrointestinal Endoscopy
Solutions to vision tasks in gastrointestinal endoscopy (GIE) conventionally use image encoders pretrained in a supervised manner with ImageNet-1k as backbones. However, the use of modern self-supervised pretraining algo…
Depth EstimationLandmark RecognitionMonocular Depth EstimationAnalyzing the Sample Complexity of Self-Supervised Image Reconstruction Methods
Supervised training of deep neural networks on pairs of clean image and noisy measurement achieves state-of-the-art performance for many image reconstruction tasks, but such training pairs are difficult to collect. Self-…
CompRess: Self-Supervised Learning by Compressing Representations
Self-supervised learning aims to learn good representations with unlabeled data. Recent works have shown that larger models benefit more from self-supervised learning than smaller models. As a result, the gap between sup…
Linear evaluationModel CompressionSelf-Supervised Image ClassificationSelf-Supervised Goal-Reaching Results in Multi-Agent Cooperation and Exploration
For groups of autonomous agents to achieve a particular goal, they must engage in coordination and long-horizon reasoning. Rather than relying on complex reward functions and explicit cooperation mechanisms, we ask what …
On the robustness of self-supervised representations for multi-view object classification
It is known that representations from self-supervised pre-training can perform on par, and often better, on various downstream tasks than representations from fully-supervised pre-training. This has been shown in a host …
Image RetrievalObjectRetrievalSemantic Segmentation