Improving Domain Generalization in Contrastive Learning using Adaptive Temperature Control
Self-supervised pre-training with contrastive learning is a powerful method for learning from sparsely labeled data. However, performance can drop considerably when there is a shift in the distribution of data from training to test time. We study this phenomenon in a setting in which the training data come from multiple domains, and the test data come from a domain not seen at training that is subject to significant covariate shift. We present a new method for contrastive learning that incorporates domain labels to increase the domain invariance of learned representations, leading to improved out-of-distribution generalization. Our method adjusts the temperature parameter in the InfoNCE loss -- which controls the relative weighting of negative pairs -- using the probability that a negative sample comes from the same domain as the anchor. This upweights pairs from more similar domains, encouraging the model to discriminate samples based on domain-invariant attributes. Through experiments on a variant of the MNIST dataset, we demonstrate that our method yields better out-of-distribution performance than domain generalization baselines. Furthermore, our method maintains strong in-distribution task performance, substantially outperforming baselines on this measure.
Code (0)
등록된 구현이 없습니다.
Tasks
Domain GeneralizationContrastive LearningSimilar Papers 제목 키워드 기반
SleepPriorCL: Contrastive Representation Learning with Prior Knowledge-based Positive Mining and Adaptive Temperature for Sleep Staging
The objective of this paper is to learn semantic representations for sleep stage classification from raw physiological time series. Although supervised methods have gained remarkable performance, they are limited in clin…
Representation LearningSelf-Supervised LearningSleep StagingTime Series+1Learning Robust Representations for Malicious Content Detection via Contrastive Sampling and Uncertainty Estimation
We propose the Uncertainty Contrastive Framework (UCF), a Positive-Unlabeled (PU) representation learning framework that integrates uncertainty-aware contrastive loss, adaptive temperature scaling, and a self-attention-g…
Representation LearningAdaptive Rigor in AI System Evaluation using Temperature-Controlled Verdict Aggregation via Generalized Power Mean
Existing evaluation methods for LLM-based AI systems, such as LLM-as-a-Judge, verdict systems, and NLI, do not always align well with human assessment because they cannot adapt their strictness to the application domain.…
Model-Aware Contrastive Learning: Towards Escaping the Dilemmas
Contrastive learning (CL) continuously achieves significant breakthroughs across multiple domains. However, the most common InfoNCE-based methods suffer from some dilemmas, such as \textit{uniformity-tolerance dilemma} (…
Contrastive LearningGraph Representation LearningRepresentation LearningSelf-Supervised Image Classification+2Dual-level Hypergraph Contrastive Learning with Adaptive Temperature Enhancement
Inspired by the success of graph contrastive learning, researchers have begun exploring the benefits of contrastive learning over hypergraphs. However, these works have the following limitations in modeling the high-orde…
Contrastive LearningHypergraph Contrastive LearningHypergraph representationsRepresentation Learning