TextTopicNet - Self-Supervised Learning of Visual Features Through Embedding Images on Semantic Text Spaces
The immense success of deep learning based methods in computer vision heavily relies on large scale training datasets. These richly annotated datasets help the network learn discriminative visual features. Collecting and annotating such datasets requires a tremendous amount of human effort and annotations are limited to popular set of classes. As an alternative, learning visual features by designing auxiliary tasks which make use of freely available self-supervision has become increasingly popular in the computer vision community. In this paper, we put forward an idea to take advantage of multi-modal context to provide self-supervision for the training of computer vision algorithms. We show that adequate visual features can be learned efficiently by training a CNN to predict the semantic textual context in which a particular image is more probable to appear as an illustration. More specifically we use popular text embedding techniques to provide the self-supervision for the training of deep CNN. Our experiments demonstrate state-of-the-art performance in image classification, object detection, and multi-modal retrieval compared to recent self-supervised or naturally-supervised approaches.
Code (1)
Tasks
image-classificationImage Classificationobject-detectionObject DetectionRetrievalSelf-Supervised LearningSimilar Papers 제목 키워드 기반
AV-DTEC: Self-Supervised Audio-Visual Fusion for Drone Trajectory Estimation and Classification
The increasing use of compact UAVs has created significant threats to public safety, while traditional drone detection systems are often bulky and costly. To address these challenges, we propose AV-DTEC, a lightweight se…
Self-Supervised LearningTrading robust representations for sample complexity through self-supervised visual experience
Learning in small sample regimes is among the most remarkable features of the human perceptual system. This ability is related to robustness to transformations, which is acquired through visual experience in the form of …
One-Shot LearningRepresentation LearningSelf-Supervised LearningSelf-supervised learning of visual features through embedding images into text topic spaces
End-to-end training from scratch of current deep architectures for new computer vision problems would require Imagenet-scale datasets, and this is not always possible. In this paper we present a method that is able to ta…
image-classificationImage Classificationobject-detectionObject Detection+2Self-Supervision on Images and Text Reduces Reliance on Visual Shortcut Features
Deep learning models trained in a fully supervised manner have been shown to rely on so-called "shortcut" features. Shortcut features are inputs that are associated with the outcome of interest in the training data, but …
Equivariant Representation Learning for Augmentation-based Self-Supervised Learning via Image Reconstruction
Augmentation-based self-supervised learning methods have shown remarkable success in self-supervised visual representation learning, excelling in learning invariant features but often neglecting equivariant ones. This li…
Image ReconstructionRepresentation LearningSelf-Supervised Learning