What Makes for Good Representations for Contrastive Learning
Contrastive learning between different views of the data achieves outstanding success in the field of self-supervised representation learning and the learned representations are useful in various downstream tasks. Since all supervision information for one view comes from the other view, contrastive learning tends to obtain the minimal sufficient representation which contains the shared information and eliminates the non-shared information between views. Considering the diversity of the downstream tasks, it can not be guaranteed that all task-relevant information is shared between views. Therefore, we assume the task-relevant information that is not shared between views can not be ignored and theoretically prove that the minimal sufficient representation in contrastive learning is not sufficient for the downstream tasks, which causes performance degradation. This reveals a new problem that the contrastive learning models have the risk of over-fitting to the shared information between views. To alleviate this problem, we propose to increase the mutual information between the representation and input as regularization to approximately introduce more task-relevant information since we can not utilize any downstream task information during training. Extensive experiments verify the rationality of our analysis and the effectiveness of our method. It significantly improves the performance of several classic contrastive learning models in downstream tasks.
Code (0)
등록된 구현이 없습니다.
Tasks
Contrastive LearningDiversityRepresentation LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
What makes instance discrimination good for transfer learning?
Contrastive visual pretraining based on the instance discrimination pretext task has made significant progress. Notably, recent work on unsupervised pretraining has shown to surpass the supervised counterpart for finetun…
object-detectionObject DetectionTransfer LearningWhat Makes Good Contrastive Learning on Small-Scale Wearable-based Tasks?
Self-supervised learning establishes a new paradigm of learning representations with much fewer or even no label annotations. Recently there has been remarkable progress on large-scale contrastive learning models which r…
Activity RecognitionContrastive LearningSelf-Supervised LearningSpatiotemporal Contrastive Video Representation Learning
We present a self-supervised Contrastive Video Representation Learning (CVRL) method to learn spatiotemporal visual representations from unlabeled videos. Our representations are learned using a contrastive loss, where t…
Action RecognitionContrastive LearningData AugmentationRepresentation Learning+4Rethinking Temperature in Graph Contrastive Learning
Due to not relying on the rare human-labeled information, self-supervised learning, especially contrastive learning, attracted much attention from researchers. It has begun to show its strong advantages on both IID data …
Contrastive LearningInductive LearningSelf-Supervised LearningRobust Contrastive Learning against Noisy Views
Contrastive learning relies on an assumption that positive pairs contain related views, e.g., patches of an image or co-occurring multimodal signals of a video, that share certain underlying information about an instance…
Binary ClassificationContrastive Learning