The NT-Xent loss upper bound
Self-supervised learning is a growing paradigm in deep representation learning, showing great generalization capabilities and competitive performance in low-labeled data regimes. The SimCLR framework proposes the NT-Xent loss for contrastive representation learning. The objective of the loss function is to maximize agreement, similarity, between sampled positive pairs. This short paper derives and proposes an upper bound for the loss and average similarity. An analysis of the implications is however not provided, but we strongly encourage anyone in the field to conduct this.
Code (0)
등록된 구현이 없습니다.
Tasks
Representation LearningSelf-Supervised LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Maximum Entropy RL (Provably) Solves Some Robust RL Problems
Many potential applications of reinforcement learning (RL) require guarantees that the agent will perform well in the face of disturbances to the dynamics or reward function. In this paper, we prove theoretically that ma…
Reinforcement Learning (RL)Cross-Entropy Games for Language Models: From Implicit Knowledge to General Capability Measures
Large Language Models (LLMs) define probability measures on text. By considering the implicit knowledge question of what it means for an LLM to know such a measure and what it entails algorithmically, we are naturally le…
Anomaly DetectioncounterfactualEstimating Mixture Entropy with Pairwise Distances
Mixture distributions arise in many parametric and non-parametric settings -- for example, in Gaussian mixture models and in non-parametric estimation. It is often necessary to compute the entropy of a mixture, but, in m…
FormMaxEnt Learners are Biased Against Giving Probability to Harmonically Bounded Candidates
AAG: Self-Supervised Representation Learning by Auxiliary Augmentation with GNT-Xent Loss
Self-supervised representation learning is an emerging research topic for its powerful capacity in learning with unlabeled data. As a mainstream self-supervised learning method, augmentation-based contrastive learning ha…
Contrastive LearningDiversityRepresentation LearningSelf-Supervised Learning