Distilling Localization for Self-Supervised Representation Learning
Recent progress in contrastive learning has revolutionized unsupervised representation learning. Concretely, multiple views (augmentations) from the same image are encouraged to map to the similar embeddings, while views from different images are pulled apart. In this paper, through visualizing and diagnosing classification errors, we observe that current contrastive models are ineffective at localizing the foreground object, limiting their ability to extract discriminative high-level features. This is due to the fact that view generation process considers pixels in an image uniformly. To address this problem, we propose a data-driven approach for learning invariance to backgrounds. It first estimates foreground saliency in images and then creates augmentations by copy-and-pasting the foreground onto a variety of backgrounds. The learning still follows the instance discrimination pretext task, so that the representation is trained to disregard background content and focus on the foreground. We study a variety of saliency estimation methods, and find that most methods lead to improvements for contrastive learning. With this approach (DiLo), significant performance is achieved for self-supervised learning on ImageNet classification, and also for object detection on PASCAL VOC and MSCOCO.
Code (0)
등록된 구현이 없습니다.
Tasks
ColorizationContrastive LearningGeneral Classificationobject-detectionObject DetectionRepresentation LearningSaliency PredictionSelf-Supervised LearningTransfer LearningVisual TrackingSimilar Papers 제목 키워드 기반
Self-Supervised Dataset Distillation for Transfer Learning
Dataset distillation methods have achieved remarkable success in distilling a large dataset into a small set of representative samples. However, they are not designed to produce a distilled dataset that can be effectivel…
Bilevel OptimizationDataset DistillationMeta-LearningRepresentation Learning+2Distilling Visual Priors from Self-Supervised Learning
Convolutional Neural Networks (CNNs) are prone to overfit small training datasets. We present a novel two-phase pipeline that leverages self-supervised learning and knowledge distillation to improve the generalization ab…
ClassificationContrastive LearningGeneral Classificationimage-classification+3DOS: Distilling Observable Softmaps of Zipfian Prototypes for Self-Supervised Point Representation
Recent advances in self-supervised learning (SSL) have shown tremendous potential for learning 3D point cloud representations without human annotations. However, SSL for 3D point clouds still faces critical challenges du…
Self-Supervised LearningSemantic Segmentation3D Object DetectionPoint CloudsDiffusion-Assisted Distillation for Self-Supervised Graph Representation Learning with MLPs
For large-scale applications, there is growing interest in replacing Graph Neural Networks (GNNs) with lightweight Multi-Layer Perceptrons (MLPs) via knowledge distillation. However, distilling GNNs for self-supervised g…
Graph Representation LearningSelf-Supervised LearningKnowledge DistillationDreamTeacher: Pretraining Image Backbones with Deep Generative Models
In this work, we introduce a self-supervised feature representation learning framework DreamTeacher that utilizes generative networks for pre-training downstream image backbones. We propose to distill knowledge from a tr…
Knowledge DistillationRepresentation Learning