Failure-Proof Non-Contrastive Self-Supervised Learning
We identify sufficient conditions to avoid known failure modes, including representation, dimensional, cluster and intracluster collapses, occurring in non-contrastive self-supervised learning. Based on these findings, we propose a principled design for the projector and loss function. We theoretically demonstrate that this design introduces an inductive bias that promotes learning representations that are both decorrelated and clustered without explicit enforcing these properties and leading to improved generalization. To the best of our knowledge, this is the first solution that achieves robust training with respect to these failure modes while guaranteeing enhanced generalization performance in downstream tasks. We validate our theoretical findings on image datasets including SVHN, CIFAR10, CIFAR100 and ImageNet-100, and show that our solution, dubbed FALCON, outperforms existing feature decorrelation and cluster-based self-supervised learning methods in terms of generalization to clustering and linear classification tasks.
Code (0)
등록된 구현이 없습니다.
Tasks
Inductive BiasSelf-Supervised LearningSimilar Papers 제목 키워드 기반
Understanding Self-supervised Contrastive Learning through Supervised Objectives
Self-supervised representation learning has achieved impressive empirical success, yet its theoretical understanding remains limited. In this work, we provide a theoretical perspective by formulating self-supervised repr…
Self-Supervised LearningRepresentation LearningContrastive LearningSelf-supervised representation learning via adaptive hard-positive mining
Despite their success in perception over the last decade, deep neural networks are also known ravenous to labeled data for training, which limits their applicability to real-world problems. Hence self-supervised learning…
Contrastive LearningRepresentation LearningSelf-Supervised LearningContrastive Learning Inverts the Data Generating Process
Contrastive learning has recently seen tremendous success in self-supervised learning. So far, however, it is largely unclear why the learned representations generalize so effectively to a large variety of downstream tas…
Contrastive LearningDisentanglementSelf-Supervised LearningLearning to Repair Lean Proofs from Compiler Feedback
As neural theorem provers become increasingly agentic, the ability to interpret and act on compiler feedback is critical. However, existing Lean datasets consist almost exclusively of correct proofs, offering little supe…
OpenCoS: Contrastive Semi-supervised Learning for Handling Open-set Unlabeled Data
Semi-supervised learning (SSL) has been a powerful strategy to incorporate few labels in learning better representations. In this paper, we focus on a practical scenario that one aims to apply SSL when unlabeled data may…
Contrastive LearningRepresentation Learning