Adversarial Momentum-Contrastive Pre-Training
Recently proposed adversarial self-supervised learning methods usually require big batches and long training epochs to extract robust features, which will bring heavy computational overhead on platforms with limited resources. In order to help the network learn more powerful feature representations in smaller batches and fewer epochs, this paper proposes a novel adversarial momentum contrastive learning method, which introduces two memory banks corresponding to clean samples and adversarial samples, respectively. These memory banks can be dynamically incorporated into the training process to track invariant features among historical mini-batches. Compared with the previous adversarial pre-training model, our method achieves superior performance with smaller batch size and less training epochs. In addition, the model outperforms some state-of-the-art supervised defensive methods on multiple benchmark datasets after being fine-tuned on downstream classification tasks.
Code (1)
Tasks
Contrastive LearningData AugmentationSelf-Supervised LearningSimilar Papers 제목 키워드 기반
ZhichunRoad at SemEval-2022 Task 2: Adversarial Training and Contrastive Learning for Multiword Representations
This paper presents our contribution to the SemEval-2022 Task 2: Multilingual Idiomaticity Detection and Sentence Embedding.We explore the impact of three different pre-trained multilingual language models in the SubTask…
Adversarial AttackContrastive LearningSentenceTask 2Momentum Contrastive Pre-training for Question Answering
Existing pre-training methods for extractive Question Answering (QA) generate cloze-like queries different from natural questions in syntax structure, which could overfit pre-trained models to simple keyword matching. In…
BenchmarkingContrastive LearningExtractive Question-AnsweringNatural Questions+1Stochastic Gradient Descent with Nonlinear Conjugate Gradient-Style Adaptive Momentum
Momentum plays a crucial role in stochastic gradient-based optimization algorithms for accelerating or improving training deep neural networks (DNNs). In deep learning practice, the momentum is usually weighted by a well…
Adversarial RobustnessA Hamiltonian Monte Carlo Method for Probabilistic Adversarial Attack and Learning
Although deep convolutional neural networks (CNNs) have demonstrated remarkable performance on multiple computer vision tasks, researches on adversarial learning have shown that deep models are vulnerable to adversarial …
Adversarial AttackImproved Baselines with Momentum Contrastive Learning
Contrastive unsupervised learning has recently shown encouraging progress, e.g., in Momentum Contrast (MoCo) and SimCLR. In this note, we verify the effectiveness of two of SimCLR's design improvements by implementing th…
Contrastive LearningData AugmentationImage ClassificationPerson Re-Identification+3