Do Self-Supervised and Supervised Methods Learn Similar Visual Representations?
Despite the success of a number of recent techniques for visual self-supervised deep learning, there has been limited investigation into the representations that are ultimately learned. By leveraging recent advances in the comparison of neural representations, we explore in this direction by comparing a contrastive self-supervised algorithm to supervision for simple image data in a common architecture. We find that the methods learn similar intermediate representations through dissimilar means, and that the representations diverge rapidly in the final few layers. We investigate this divergence, finding that these layers strongly fit to their distinct learning objectives. We also find that the contrastive objective implicitly fits the supervised objective in intermediate layers, but that the reverse is not true. Our work particularly highlights the importance of the learned intermediate representations, and raises critical questions for auxiliary task design.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Self-similarity Prior Distillation for Unsupervised Remote Physiological Measurement
Remote photoplethysmography (rPPG) is a noninvasive technique that aims to capture subtle variations in facial pixels caused by changes in blood volume resulting from cardiac activities. Most existing unsupervised method…
Contrastive LearningSS-BSN: Attentive Blind-Spot Network for Self-Supervised Denoising with Nonlocal Self-Similarity
Recently, numerous studies have been conducted on supervised learning-based image denoising methods. However, these methods rely on large-scale noisy-clean image pairs, which are difficult to obtain in practice. Denoisin…
DenoisingImage DenoisingImage RestorationSeries2Vec: Similarity-based Self-supervised Representation Learning for Time Series Classification
We argue that time series analysis is fundamentally different in nature to either vision or natural language processing with respect to the forms of meaningful self-supervised learning tasks that can be defined. Motivate…
Data AugmentationRepresentation LearningSelf-Supervised LearningTime Series+2Co-learning: Learning from Noisy Labels with Self-supervision
Noisy labels, resulting from mistakes in manual labeling or webly data collecting for supervised learning, can cause neural networks to overfit the misleading information and degrade the generalization performance. Self-…
Learning with noisy labelsSelf-Supervised LearningOn the surprising similarities between supervised and self-supervised models
How do humans learn to acquire a powerful, flexible and robust representation of objects? While much of this process remains unknown, it is clear that humans do not require millions of object labels. Excitingly, recent a…
ObjectObject RecognitionSelf-Supervised Learning