Unveiling the Potential of Probabilistic Embeddings in Self-Supervised Learning
In recent years, self-supervised learning has played a pivotal role in advancing machine learning by allowing models to acquire meaningful representations from unlabeled data. An intriguing research avenue involves developing self-supervised models within an information-theoretic framework, but many studies often deviate from the stochasticity assumptions made when deriving their objectives. To gain deeper insights into this issue, we propose to explicitly model the representation with stochastic embeddings and assess their effects on performance, information compression and potential for out-of-distribution detection. From an information-theoretic perspective, we seek to investigate the impact of probabilistic modeling on the information bottleneck, shedding light on a trade-off between compression and preservation of information in both representation and loss space. Emphasizing the importance of distinguishing between these two spaces, we demonstrate how constraining one can affect the other, potentially leading to performance degradation. Moreover, our findings suggest that introducing an additional bottleneck in the loss space can significantly enhance the ability to detect out-of-distribution examples, only leveraging either representation features or the variance of their underlying distribution.
Code (0)
등록된 구현이 없습니다.
Tasks
Out-of-Distribution DetectionSelf-Supervised LearningSimilar Papers 제목 키워드 기반
Unveiling Interpretability in Self-Supervised Speech Representations for Parkinson's Diagnosis
Recent works in pathological speech analysis have increasingly relied on powerful self-supervised speech representations, leading to promising results. However, the complex, black-box nature of these embeddings and the l…
A Probabilistic Approach to Self-Supervised Learning using Cyclical Stochastic Gradient MCMC
In this paper we present a practical Bayesian self-supervised learning method with Cyclical Stochastic Gradient Hamiltonian Monte Carlo (cSGHMC). Within this framework, we place a prior over the parameters of a self-supe…
Out-of-Distribution DetectionSelf-Supervised LearningSelf-supervised learning of class embeddings from video
This work explores how to use self-supervised learning on videos to learn a class-specific image embedding that encodes pose and shape information. At train time, two frames of the same video of an object class (e.g. hum…
DecoderSelf-Supervised LearningProbabilistic Contrastive Loss for Self-Supervised Learning
This paper proposes a probabilistic contrastive loss function for self-supervised learning. The well-known contrastive loss is deterministic and involves a temperature hyperparameter that scales the inner product between…
Contrastive LearningSelf-Supervised LearningOn minimal variations for unsupervised representation learning
Unsupervised representation learning aims at describing raw data efficiently to solve various downstream tasks. It has been approached with many techniques, such as manifold learning, diffusion maps, or more recently sel…
Representation LearningSelf-Supervised Learning