Generalization Analysis for Contrastive Representation Learning
Recently, contrastive learning has found impressive success in advancing the state of the art in solving various machine learning tasks. However, the existing generalization analysis is very limited or even not meaningful. In particular, the existing generalization error bounds depend linearly on the number $k$ of negative examples while it was widely shown in practice that choosing a large $k$ is necessary to guarantee good generalization of contrastive learning in downstream tasks. In this paper, we establish novel generalization bounds for contrastive learning which do not depend on $k$, up to logarithmic terms. Our analysis uses structural results on empirical covering numbers and Rademacher complexities to exploit the Lipschitz continuity of loss functions. For self-bounding Lipschitz loss functions, we further improve our results by developing optimistic bounds which imply fast rates in a low noise condition. We apply our results to learning with both linear representation and nonlinear representation by deep neural networks, for both of which we derive Rademacher complexity bounds to get improved generalization bounds.
Code (0)
등록된 구현이 없습니다.
Tasks
Contrastive LearningGeneralization BoundsRepresentation LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Using contrastive learning to improve the performance of steganalysis schemes
To improve the detection accuracy and generalization of steganalysis, this paper proposes the Steganalysis Contrastive Framework (SCF) based on contrastive learning. The SCF improves the feature representation of stegana…
Contrastive LearningSteganalysisGeneralization Analysis for Contrastive Representation Learning under Non-IID Settings
Contrastive Representation Learning (CRL) has achieved impressive success in various domains in recent years. Nevertheless, the theoretical understanding of the generalization behavior of CRL is limited. Moreover, to the…
Contrastive LearningGeneralization BoundsRepresentation LearningTowards Understanding the Mechanism of Contrastive Learning via Similarity Structure: A Theoretical Analysis
Contrastive learning is an efficient approach to self-supervised representation learning. Although recent studies have made progress in the theoretical understanding of contrastive learning, the investigation of how to c…
Contrastive LearningRepresentation LearningRevisiting Theory of Contrastive Learning for Domain Generalization
Contrastive learning is among the most popular and powerful approaches for self-supervised representation learning, where the goal is to map semantically similar samples close together while separating dissimilar ones in…
Representation LearningDomain GeneralizationContrastive LearningGeneralization Beyond Feature Alignment: Concept Activation-Guided Contrastive Learning
Learning invariant representations via contrastive learning has seen state-of-the-art performance in domain generalization (DG). Despite such success, in this paper, we find that its core learning strategy -- feature ali…
Contrastive LearningDiversityDomain Generalization