The SSL Interplay: Augmentations, Inductive Bias, and Generalization
Self-supervised learning (SSL) has emerged as a powerful framework to learn representations from raw data without supervision. Yet in practice, engineers face issues such as instability in tuning optimizers and collapse of representations during training. Such challenges motivate the need for a theory to shed light on the complex interplay between the choice of data augmentation, network architecture, and training algorithm. We study such an interplay with a precise analysis of generalization performance on both pretraining and downstream tasks in a theory friendly setup, and highlight several insights for SSL practitioners that arise from our theory.
Code (0)
등록된 구현이 없습니다.
Tasks
Data AugmentationInductive BiasSelf-Supervised LearningSimilar Papers 제목 키워드 기반
Understanding Contrastive Learning Requires Incorporating Inductive Biases
Contrastive learning is a popular form of self-supervised learning that encourages augmentations (views) of the same input to have more similar representations compared to augmentations of different inputs. Recent attemp…
Contrastive LearningSelf-Supervised LearningIdentity Crisis: Memorization and Generalization under Extreme Overparameterization
We study the interplay between memorization and generalization of overparameterized networks in the extreme case of a single training example and an identity-mapping task. We examine fully-connected and convolutional net…
MemorizationEquivariant score-based generative models provably learn distributions with symmetries efficiently
Symmetry is ubiquitous in many real-world phenomena and tasks, such as physics, images, and molecular simulations. Empirical studies have demonstrated that incorporating symmetries into generative models can provide bett…
Data AugmentationGeneralization BoundsInductive BiasStatistical Properties of Training & Generalization
Deep learning has managed to evade numerous intuitions from classical statistics to achieve unprecedented performance on a number of real-world tasks. In this article, we investigate the key features and surprises of dee…
A Random CNN Sees Objects: One Inductive Bias of CNN and Its Applications
This paper starts by revealing a surprising finding: without any learning, a randomly initialized CNN can localize objects surprisingly well. That is, a CNN has an inductive bias to naturally focus on objects, named as T…
Inductive BiasObjectobject-detectionObject Detection+1