Understanding Self-Supervised Learning via Latent Distribution Matching
Self-supervised learning (SSL) excels at finding general-purpose latent representations from complex data, yet lacks a unifying theoretical framework that explains the diverse existing methods and guides the design of new ones. We cast SSL as latent distribution matching (LDM): learning representations that maximize their log-probability under an assumed latent model (alignment), while maximizing latent entropy to prevent collapse (uniformity). This view unifies independent component analysis with contrastive, non-contrastive, and predictive SSL methods, including stop gradient approaches. Leveraging LDM, we derive a nonlinear, sampling-free Bayesian filtering model with a Kalman-based predictor for high-dimensional timeseries. We further prove that predictive LDM yields identifiable latent representations under mild assumptions, even with nonlinear predictors. Overall, LDM clarifies the assumptions behind established SSL methods and provides principled guidance for developing new approaches.
Code (0)
등록된 구현이 없습니다.
Tasks
Self-Supervised LearningSimilar Papers 제목 키워드 기반
Momentum Contrastive Autoencoder: Using Contrastive Learning for Latent Space Distribution Matching in WAE
Wasserstein autoencoder (WAE) shows that matching two distributions is equivalent to minimizing a simple autoencoder (AE) loss under the constraint that the latent space of this AE matches a pre-specified prior distribut…
Contrastive LearningRepresentation LearningDistribution Matching Variational AutoEncoder
Most visual generative models compress images into a latent space before applying diffusion or autoregressive modelling. Yet, existing approaches such as VAEs and foundation model aligned encoders implicitly constrain th…
Momentum Contrastive Autoencoder
Wasserstein autoencoder (WAE) shows that matching two distributions is equivalent to minimizing a simple autoencoder (AE) loss under the constraint that the latent space of this AE matches a pre-specified prior distribut…
Contrastive LearningRepresentation LearningUnderstanding Self-supervised Learning via Information Bottleneck Principle
Self-supervised learning alleviates the massive demands for annotations in deep learning, and recent advances are mainly dominated by contrastive learning. Existed contrastive learning methods narrows the distance betwee…
Contrastive LearningSelf-Supervised LearningSemi-Supervised Generative Learning via Latent Space Distribution Matching
We introduce Latent Space Distribution Matching (LSDM), a novel framework for semi-supervised generative modeling of conditional distributions. LSDM operates in two stages: (i) learning a low-dimensional latent space fro…
Image Super-Resolution