On Compositions of Transformations in Contrastive Self-Supervised Learning
In the image domain, excellent representations can be learned by inducing invariance to content-preserving transformations via noise contrastive learning. In this paper, we generalize contrastive learning to a wider set of transformations, and their compositions, for which either invariance or distinctiveness is sought. We show that it is not immediately obvious how existing methods such as SimCLR can be extended to do so. Instead, we introduce a number of formal requirements that all contrastive formulations must satisfy, and propose a practical construction which satisfies these requirements. In order to maximise the reach of this analysis, we express all components of noise contrastive formulations as the choice of certain generalized transformations of the data (GDTs), including data sampling. We then consider videos as an example of data in which a large variety of transformations are applicable, accounting for the extra modalities -- for which we analyze audio and text -- and the dimension of time. We find that being invariant to certain transformations and distinctive to others is critical to learning effective video representations, improving the state-of-the-art for multiple benchmarks by a large margin, and even surpassing supervised pretraining.
Code (1)
Tasks
Action RecognitionAudio ClassificationContrastive LearningRepresentation LearningSelf-Supervised LearningSimilar Papers 제목 키워드 기반
Self-Supervised Learning for Group Equivariant Neural Networks
This paper proposes a method to construct pretext tasks for self-supervised learning on group equivariant neural networks. Group equivariant neural networks are the models whose structure is restricted to commute with th…
Self-Supervised LearningCan Temporal Information Help with Contrastive Self-Supervised Learning?
Leveraging temporal information has been regarded as essential for developing video understanding models. However, how to properly incorporate temporal information into the recent successful instance discrimination based…
Data AugmentationRepresentation LearningSelf-Supervised LearningVideo UnderstandingThe Impact of Spatiotemporal Augmentations on Self-Supervised Audiovisual Representation Learning
Contrastive learning of auditory and visual perception has been extremely successful when investigated individually. However, there are still major questions on how we could integrate principles learned from both domains…
Contrastive LearningRepresentation LearningSelf-Supervised LearningTowards Robust Graph Contrastive Learning
We study the problem of adversarially robust self-supervised learning on graphs. In the contrastive learning framework, we introduce a new method that increases the adversarial robustness of the learned representations t…
Adversarial RobustnessContrastive LearningSelf-Supervised LearningTime-Equivariant Contrastive Video Representation Learning
We introduce a novel self-supervised contrastive learning method to learn representations from unlabelled videos. Existing approaches ignore the specifics of input distortions, e.g., by learning invariance to temporal tr…
Action RecognitionContrastive LearningRepresentation LearningRetrieval+1