How Does SimSiam Avoid Collapse Without Negative Samples? A Unified Understanding with Self-supervised Contrastive Learning
To avoid collapse in self-supervised learning (SSL), a contrastive loss is widely used but often requires a large number of negative samples. Without negative samples yet achieving competitive performance, a recent work has attracted significant attention for providing a minimalist simple Siamese (SimSiam) method to avoid collapse. However, the reason for how it avoids collapse without negative samples remains not fully clear and our investigation starts by revisiting the explanatory claims in the original SimSiam. After refuting their claims, we introduce vector decomposition for analyzing the collapse based on the gradient analysis of the $l_2$-normalized representation vector. This yields a unified perspective on how negative samples and SimSiam alleviate collapse. Such a unified perspective comes timely for understanding the recent progress in SSL.
Code (0)
등록된 구현이 없습니다.
Tasks
Contrastive LearningSelf-Supervised LearningSimilar Papers 제목 키워드 기반
How Does SimSiam Avoid Collapse Without Negative Samples? Towards a Unified Understanding of Progress in SSL
Towards avoiding collapse in self-supervised learning (SSL), contrastive loss is widely used but often requires a large number of negative samples. Without negative samples yet achieving competitive performance one recen…
Self-Supervised LearningUnderstanding Collapse in Non-Contrastive Siamese Representation Learning
Contrastive methods have led a recent surge in the performance of self-supervised representation learning (SSL). Recent methods like BYOL or SimSiam purportedly distill these contrastive methods down to their essence, re…
Continual LearningContrastive LearningRepresentation LearningFeature Normalization Prevents Collapse of Non-contrastive Learning Dynamics
Contrastive learning is a self-supervised representation learning framework, where two positive views generated through data augmentation are made similar by an attraction force in a data representation space, while a re…
Computational EfficiencyContrastive LearningData AugmentationRepresentation LearningImplicit variance regularization in non-contrastive SSL
Non-contrastive SSL methods like BYOL and SimSiam rely on asymmetric predictor networks to avoid representational collapse without negative samples. Yet, how predictor networks facilitate stable learning is not fully und…
Understanding self-supervised Learning Dynamics without Contrastive Pairs
While contrastive approaches of self-supervised learning (SSL) learn representations by minimizing the distance between two augmented views of the same data point (positive pairs) and maximizing views from different data…
Self-Supervised Learning