Max-Margin Contrastive Learning
Standard contrastive learning approaches usually require a large number of negatives for effective unsupervised learning and often exhibit slow convergence. We suspect this behavior is due to the suboptimal selection of negatives used for offering contrast to the positives. We counter this difficulty by taking inspiration from support vector machines (SVMs) to present max-margin contrastive learning (MMCL). Our approach selects negatives as the sparse support vectors obtained via a quadratic optimization problem, and contrastiveness is enforced by maximizing the decision margin. As SVM optimization can be computationally demanding, especially in an end-to-end setting, we present simplifications that alleviate the computational burden. We validate our approach on standard vision benchmark datasets, demonstrating better performance in unsupervised representation learning over state-of-the-art, while having better empirical convergence properties.
Code (1)
Tasks
Contrastive LearningRepresentation LearningSelf-Supervised Image ClassificationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Understanding Contrastive Learning Through the Lens of Margins
Contrastive learning, along with its variations, has been a highly effective self-supervised learning method across diverse domains. Contrastive learning measures the distance between representations using cosine similar…
Contrastive LearningRepresentation LearningSelf-Supervised LearningSpeaker RecognitionMargin Preserving Self-paced Contrastive Learning Towards Domain Adaptation for Medical Image Segmentation
To bridge the gap between the source and target domains in unsupervised domain adaptation (UDA), the most common strategy puts focus on matching the marginal distributions in the feature space through adversarial learnin…
Cardiac SegmentationContrastive LearningDomain AdaptationImage Segmentation+4Experimenting with Additive Margins for Contrastive Self-Supervised Speaker Verification
Most state-of-the-art self-supervised speaker verification systems rely on a contrastive-based objective function to learn speaker representations from unlabeled speech data. We explore different ways to improve the perf…
Speaker VerificationeMargin: Revisiting Contrastive Learning with Margin-Based Separation
We revisit previous contrastive learning frameworks to investigate the effect of introducing an adaptive margin into the contrastive loss function for time series representation learning. Specifically, we explore whether…
Representation LearningContrastive LearningEnhancing Adversarial Robustness of Deep Neural Networks Through Supervised Contrastive Learning
Adversarial attacks exploit the vulnerabilities of convolutional neural networks by introducing imperceptible perturbations that lead to misclassifications, exposing weaknesses in feature representations and decision bou…
Adversarial RobustnessContrastive Learning