Improved Baselines with Momentum Contrastive Learning
Contrastive unsupervised learning has recently shown encouraging progress, e.g., in Momentum Contrast (MoCo) and SimCLR. In this note, we verify the effectiveness of two of SimCLR's design improvements by implementing them in the MoCo framework. With simple modifications to MoCo---namely, using an MLP projection head and more data augmentation---we establish stronger baselines that outperform SimCLR and do not require large training batches. We hope this will make state-of-the-art unsupervised learning research more accessible. Code will be made public.
Code (36)
Tasks
Contrastive LearningData AugmentationImage ClassificationPerson Re-IdentificationRepresentation LearningSelf-Supervised Image ClassificationSelf-Supervised Person Re-IdentificationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Momentum Contrastive Pre-training for Question Answering
Existing pre-training methods for extractive Question Answering (QA) generate cloze-like queries different from natural questions in syntax structure, which could overfit pre-trained models to simple keyword matching. In…
BenchmarkingContrastive LearningExtractive Question-AnsweringNatural Questions+1CoCoSoDa: Effective Contrastive Learning for Code Search
Code search aims to retrieve semantically relevant code snippets for a given natural language query. Recently, many approaches employing contrastive learning have shown promising results on code representation learning a…
Code SearchContrastive LearningData AugmentationRepresentation LearningSelf-supervised Text-independent Speaker Verification using Prototypical Momentum Contrastive Learning
In this study, we investigate self-supervised representation learning for speaker verification (SV). First, we examine a simple contrastive learning approach (SimCLR) with a momentum contrastive (MoCo) learning framework…
ClusteringContrastive LearningRepresentation LearningSpeaker Verification+1AMD: Adaptive Momentum and Decoupled Contrastive Learning Framework for Robust Long-Tail Trajectory Prediction
Accurately predicting the future trajectories of traffic agents is essential in autonomous driving. However, due to the inherent imbalance in trajectory distributions, tail data in natural datasets often represents more …
Trajectory PredictionContrastive LearningAutonomous DrivingSelf-Supervised Visual Representation Learning via Residual Momentum
Self-supervised learning (SSL) approaches have shown promising capabilities in learning the representation from unlabeled data. Amongst them, momentum-based frameworks have attracted significant attention. Despite being …
Contrastive LearningRepresentation LearningSelf-Supervised Learning