TiCo: Transformation Invariance and Covariance Contrast for Self-Supervised Visual Representation Learning
We present Transformation Invariance and Covariance Contrast (TiCo) for self-supervised visual representation learning. Similar to other recent self-supervised learning methods, our method is based on maximizing the agreement among embeddings of different distorted versions of the same image, which pushes the encoder to produce transformation invariant representations. To avoid the trivial solution where the encoder generates constant vectors, we regularize the covariance matrix of the embeddings from different images by penalizing low rank solutions. By jointly minimizing the transformation invariance loss and covariance contrast loss, we get an encoder that is able to produce useful representations for downstream tasks. We analyze our method and show that it can be viewed as a variant of MoCo with an implicit memory bank of unlimited size at no extra memory cost. This makes our method perform better than alternative methods when using small batch sizes. TiCo can also be seen as a modification of Barlow Twins. By connecting the contrastive and redundancy-reduction methods together, TiCo gives us new insights into how joint embedding methods work.
Code (2)
Tasks
Representation LearningSelf-Supervised LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Learning Online Visual Invariances for Novel Objects via Supervised and Self-Supervised Training
Humans can identify objects following various spatial transformations such as scale and viewpoint. This extends to novel objects, after a single presentation at a single pose, sometimes referred to as online invariance. …
Data AugmentationTranslationOn Compositions of Transformations in Contrastive Self-Supervised Learning
In the image domain, excellent representations can be learned by inducing invariance to content-preserving transformations via noise contrastive learning. In this paper, we generalize contrastive learning to a wider set …
Action RecognitionAudio ClassificationContrastive LearningRepresentation Learning+1Time-Equivariant Contrastive Video Representation Learning
We introduce a novel self-supervised contrastive learning method to learn representations from unlabelled videos. Existing approaches ignore the specifics of input distortions, e.g., by learning invariance to temporal tr…
Action RecognitionContrastive LearningRepresentation LearningRetrieval+1Explicit homography estimation improves contrastive self-supervised learning
The typical contrastive self-supervised algorithm uses a similarity measure in latent space as the supervision signal by contrasting positive and negative images directly or indirectly. Although the utility of self-super…
Contrastive LearningHomography EstimationSelf-Supervised LearningImproving Transformation Invariance in Contrastive Representation Learning
We propose methods to strengthen the invariance properties of representations obtained by contrastive learning. While existing approaches implicitly induce a degree of invariance as representations are learned, we look t…
Contrastive LearningRepresentation Learning