Self-supervised Pre-training with Hard Examples Improves Visual Representations
Self-supervised pre-training (SSP) employs random image transformations to generate training data for visual representation learning. In this paper, we first present a modeling framework that unifies existing SSP methods as learning to predict pseudo-labels. Then, we propose new data augmentation methods of generating training examples whose pseudo-labels are harder to predict than those generated via random image transformations. Specifically, we use adversarial training and CutMix to create hard examples (HEXA) to be used as augmented views for MoCo-v2 and DeepCluster-v2, leading to two variants HEXA_{MoCo} and HEXA_{DCluster}, respectively. In our experiments, we pre-train models on ImageNet and evaluate them on multiple public benchmarks. Our evaluation shows that the two new algorithm variants outperform their original counterparts, and achieve new state-of-the-art on a wide range of tasks where limited task supervision is available for fine-tuning. These results verify that hard examples are instrumental in improving the generalization of the pre-trained models.
Code (0)
등록된 구현이 없습니다.
Tasks
Data AugmentationRepresentation LearningSelf-Supervised Image ClassificationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Self-supervised Adversarial Training
Recent work has demonstrated that neural networks are vulnerable to adversarial examples. To escape from the predicament, many works try to harden the model in various ways, in which adversarial training is an effective …
Self-Supervised LearningSelf-Supervised Contrastive Learning with Adversarial Perturbations for Defending Word Substitution-based Attacks
In this paper, we present an approach to improve the robustness of BERT language models against word substitution-based adversarial attacks by leveraging adversarial perturbations for self-supervised contrastive learning…
Adversarial AttackContrastive LearningLanguage ModellingAdversarial Contrastive Self-Supervised Learning
Recently, learning from vast unlabeled data, especially self-supervised learning, has been emerging and attracted widespread attention. Self-supervised learning followed by the supervised fine-tuning on a few labeled exa…
Self-Supervised LearningTripletContrastive Learning with Adversarial Examples
Contrastive learning (CL) is a popular technique for self-supervised learning (SSL) of visual representations. It uses pairs of augmentations of unlabeled training examples to define a classification task for pretext lea…
Contrastive LearningSelf-Supervised LearningA Survey of the Impact of Self-Supervised Pretraining for Diagnostic Tasks with Radiological Images
Self-supervised pretraining has been observed to be effective at improving feature representations for transfer learning, leveraging large amounts of unlabelled data. This review summarizes recent research into its usage…
Clinical KnowledgeDiagnosticSelf-Supervised LearningTransfer Learning