Improving the Transferability of Supervised Pretraining with an MLP Projector
The pretrain-finetune paradigm is a classical pipeline in visual learning. Recent progress on unsupervised pretraining methods showed superior transfer performance to their supervised counterparts. While a few works attempted to explore the underlying mechanisms, the reasons behind the transferability gaps still have not been fully explored. This paper reveals that the multilayer perceptron (MLP) projector is a key factor for the better transferability of unsupervised pretraining. Based on this observation, we attempt to close the transferability gap between supervised pretraining and unsupervised pretraining by adding an MLP projector before the classifier of supervised pretraining. Our analysis indicates that the MLP projector can help retain intra-class variation of visual features, decrease the feature distribution distance between pretraining dataset and evaluation dataset, and reduce feature redundancy for effective adaptation to new tasks. Extensive experiments demonstrate that the added MLP projector significantly boosts the transferability of supervised pretraining, \emph{e.g.,} \textbf{+7.2\%} top-1 accuracy on the unseen class generalization task and \textbf{+5.7\%} top-1 accuracy on 12-domain classification tasks, making supervised pretraining even better than unsupervised pretraining.
Code (0)
등록된 구현이 없습니다.
Tasks
domain classificationSimilar Papers 제목 키워드 기반
Revisiting the Transferability of Supervised Pretraining: an MLP Perspective
The pretrain-finetune paradigm is a classical pipeline in visual learning. Recent progress on unsupervised pretraining methods shows superior transfer performance to their supervised counterparts. This paper revisits thi…
domain classificationimage-classificationImage ClassificationLinear evaluation+3Revisiting Supervision for Continual Representation Learning
In the field of continual learning, models are designed to learn tasks one after the other. While most research has centered on supervised continual learning, there is a growing interest in unsupervised continual learnin…
Continual LearningRepresentation LearningSelf-Supervised LearningA surprisingly simple technique to control the pretraining bias for better transfer: Expand or Narrow your representation
Self-Supervised Learning (SSL) models rely on a pretext task to learn representations. Because this pretext task differs from the downstream tasks used to evaluate the performance of these models, there is an inherent mi…
Self-Supervised LearningAugmentation-aware Self-supervised Learning with Conditioned Projector
Self-supervised learning (SSL) is a powerful technique for learning from unlabeled data. By learning to remain invariant to applied data augmentations, methods such as SimCLR and MoCo can reach quality on par with superv…
Self-Supervised LearningSensitivityNo Reason for No Supervision: Improved Generalization in Supervised Models
We consider the problem of training a deep neural network on a given classification task, e.g., ImageNet-1K (IN1K), so that it excels at both the training task as well as at other (future) transfer tasks. These two seemi…
Data AugmentationSelf-Supervised LearningTransfer Learning