Conditional Alignment and Uniformity for Contrastive Learning with Continuous Proxy Labels
Contrastive Learning has shown impressive results on natural and medical images, without requiring annotated data. However, a particularity of medical images is the availability of meta-data (such as age or sex) that can be exploited for learning representations. Here, we show that the recently proposed contrastive y-Aware InfoNCE loss, that integrates multi-dimensional meta-data, asymptotically optimizes two properties: conditional alignment and global uniformity. Similarly to [Wang, 2020], conditional alignment means that similar samples should have similar features, but conditionally on the meta-data. Instead, global uniformity means that the (normalized) features should be uniformly distributed on the unit hyper-sphere, independently of the meta-data. Here, we propose to define conditional uniformity, relying on the meta-data, that repel only samples with dissimilar meta-data. We show that direct optimization of both conditional alignment and uniformity improves the representations, in terms of linear evaluation, on both CIFAR-100 and a brain MRI dataset.
Code (0)
등록된 구현이 없습니다.
Tasks
Contrastive LearningLinear evaluationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Following the Autoregressive Nature of LLM Embeddings via Compression and Alignment
A new trend uses LLMs as dense text encoders via contrastive learning. However, since LLM embeddings predict the probability distribution of the next token, they are inherently generative and distributive, conflicting wi…
Contrastive LearningD2CSE: Difference-aware Deep continuous prompts for Contrastive Sentence Embeddings
This paper describes Difference-aware Deep continuous prompt for Contrastive Sentence Embeddings (D2CSE) that learns sentence embeddings. Compared to state-of-the-art approaches, D2CSE computes sentence vectors that are …
Contrastive LearningRetrievalSemantic Textual SimilaritySentence+2Rethinking Graph Masked Autoencoders through Alignment and Uniformity
Self-supervised learning on graphs can be bifurcated into contrastive and generative methods. Contrastive methods, also known as graph contrastive learning (GCL), have dominated graph self-supervised learning in the past…
Contrastive LearningSelf-Supervised LearningWhitening-based Contrastive Learning of Sentence Embeddings
This paper presents a whitening-based contrastive learning method for sentence embedding learning (WhitenedCSE), which combines contrastive learning with a novel shuffled group whitening. Generally, contrastive learning …
Contrastive LearningDiversitySemantic Textual SimilaritySentence+4Correlation between Alignment-Uniformity and Performance of Dense Contrastive Representations
Recently, dense contrastive learning has shown superior performance on dense prediction tasks compared to instance-level contrastive learning. Despite its supremacy, the properties of dense contrastive representations ha…
Contrastive LearningLinear evaluation