Contrastive Learning with Stronger Augmentations
Representation learning has significantly been developed with the advance of contrastive learning methods. Most of those methods have benefited from various data augmentations that are carefully designated to maintain their identities so that the images transformed from the same instance can still be retrieved. However, those carefully designed transformations limited us to further explore the novel patterns exposed by other transformations. Meanwhile, as found in our experiments, the strong augmentations distorted the images' structures, resulting in difficult retrieval. Thus, we propose a general framework called Contrastive Learning with Stronger Augmentations~(CLSA) to complement current contrastive learning approaches. Here, the distribution divergence between the weakly and strongly augmented images over the representation bank is adopted to supervise the retrieval of strongly augmented queries from a pool of instances. Experiments on the ImageNet dataset and downstream datasets showed the information from the strongly augmented images can significantly boost the performance. For example, CLSA achieves top-1 accuracy of 76.2% on ImageNet with a standard ResNet-50 architecture with a single-layer classifier fine-tuned, which is almost the same level as 76.5% of supervised results. The code and pre-trained models are available in https://github.com/maple-research-lab/CLSA.
Code (1)
Tasks
Contrastive LearningRepresentation LearningRetrievalSelf-Supervised Image ClassificationSelf-Supervised LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Stronger is not better: Better Augmentations in Contrastive Learning for Medical Image Segmentation
Self-supervised contrastive learning is among the recent representation learning methods that have shown performance gains in several downstream tasks including semantic segmentation. This paper evaluates strong data aug…
Medical Image SegmentationRepresentation LearningSemantic SegmentationContrastive LearningTraining GANs with Stronger Augmentations via Contrastive Discriminator
Recent works in Generative Adversarial Networks (GANs) are actively revisiting various data augmentation techniques as an effective way to prevent discriminator overfitting. It is still unclear, however, that which augme…
Contrastive LearningData AugmentationLinear evaluationRepresentation LearningStrongly Augmented Contrastive Clustering
Deep clustering has attracted increasing attention in recent years due to its capability of joint representation learning and clustering via deep neural networks. In its latest developments, the contrastive learning has …
ClusteringContrastive LearningDeep ClusteringRepresentation LearningToward Understanding the Feature Learning Process of Self-supervised Contrastive Learning
How can neural networks trained by contrastive learning extract features from the unlabeled data? Why does contrastive learning usually need much stronger data augmentations than supervised learning to ensure good repres…
Contrastive LearningSelf-Supervised LearningThe Impact of Spatiotemporal Augmentations on Self-Supervised Audiovisual Representation Learning
Contrastive learning of auditory and visual perception has been extremely successful when investigated individually. However, there are still major questions on how we could integrate principles learned from both domains…
Contrastive LearningRepresentation LearningSelf-Supervised Learning