PatchRot: A Self-Supervised Technique for Training Vision Transformers
Vision transformers require a huge amount of labeled data to outperform convolutional neural networks. However, labeling a huge dataset is a very expensive process. Self-supervised learning techniques alleviate this problem by learning features similar to supervised learning in an unsupervised way. In this paper, we propose a self-supervised technique PatchRot that is crafted for vision transformers. PatchRot rotates images and image patches and trains the network to predict the rotation angles. The network learns to extract both global and local features from an image. Our extensive experiments on different datasets showcase PatchRot training learns rich features which outperform supervised learning and compared baseline.
Code (1)
Tasks
Self-Supervised LearningSimilar Papers 제목 키워드 기반
Boosting Supervision with Self-Supervision for Few-shot Learning
We present a technique to improve the transferability of deep representations learned on small labeled datasets by introducing self-supervised tasks as auxiliary loss functions. While recent approaches for self-supervise…
Few-Shot LearningSelf-Supervised LearningImproved Self-Supervised Deep Image Denoising
We describe techniques for training high-quality image denoising models that require only single instances of corrupted images as training data. Inspired by a recent technique that removes the need for supervision throug…
DenoisingImage DenoisingInvestigating Zero-Shot Generalizability on Mandarin-English Code-Switched ASR and Speech-to-text Translation of Recent Foundation Models with Self-Supervision and Weak Supervision
This work evaluated several cutting-edge large-scale foundation models based on self-supervision or weak supervision, including SeamlessM4T, SeamlessM4T v2, and Whisper-large-v3, on three code-switched corpora. We found …
Speech-to-TextSpeech-to-Text TranslationTextTopicNet - Self-Supervised Learning of Visual Features Through Embedding Images on Semantic Text Spaces
The immense success of deep learning based methods in computer vision heavily relies on large scale training datasets. These richly annotated datasets help the network learn discriminative visual features. Collecting and…
image-classificationImage Classificationobject-detectionObject Detection+2Advancing Melanoma Diagnosis with Self-Supervised Neural Networks: Evaluating the Effectiveness of Different Techniques
We investigate the potential of self-supervision in improving the accuracy of deep learning models trained to classify melanoma patches. Various self-supervision techniques such as rotation prediction, missing patch pred…
Contrastive LearningMelanoma Diagnosis