Multi-Task Self-Training for Learning General Representations

2021-08-25 · ICCV 2021 10 · Golnaz Ghiasi, Barret Zoph, Ekin D. Cubuk, Quoc V. Le, Tsung-Yi Lin

Despite the fast progress in training specialized models for various tasks, learning a single general model that works well for many tasks is still challenging for computer vision. Here we introduce multi-task self-training (MuST), which harnesses the knowledge in independent specialized teacher models (e.g., ImageNet model on classification) to train a single general student model. Our approach has three steps. First, we train specialized teachers independently on labeled datasets. We then use the specialized teachers to label an unlabeled dataset to create a multi-task pseudo labeled dataset. Finally, the dataset, which now contains pseudo labels from teacher models trained on different datasets/tasks, is then used to train a student model with multi-task learning. We evaluate the feature representations of the student model on 6 vision tasks including image recognition (classification, detection, segmentation)and 3D geometry estimation (depth and surface normal estimation). MuST is scalable with unlabeled or partially labeled datasets and outperforms both specialized supervised models and self-supervised models when training on large scale datasets. Lastly, we show MuST can improve upon already strong checkpoints trained with billions of examples. The results suggest self-training is a promising direction to aggregate labeled and unlabeled training data for learning general feature representations.

📄 PDF Abstract BibTeX arXiv:2108.11353

Code (0)

등록된 구현이 없습니다.

Tasks

3D geometryMulti-Task LearningPartially Labeled DatasetsSurface Normal Estimation

Improving out-of-distribution generalization via multi-task self-supervised pretraining

2020-03-30 · Isabela Albuquerque, Nikhil Naik, Junnan Li, Nitish Keskar 외

Self-supervised feature representations have been shown to be useful for supervised classification, few-shot learning, and adversarial robustness. We show that features obtained using self-supervised learning are compara…

Adversarial Robustness Domain Generalization Few-Shot Learning Multi-Task Learning+2

Domain-aware Self-supervised Pre-training for Label-Efficient Meme Analysis

2022-09-29 · Shivam Sharma, Mohd Khizir Siddiqui, Md. Shad Akhtar, Tanmoy Chakraborty

Existing self-supervised learning strategies are constrained to either a limited set of objectives or generic downstream tasks that predominantly target uni-modal applications. This has isolated progress for imperative m…

Representation Learning Self-Supervised Learning

Learning Generalizable 3D Medical Image Representations from Mask-Guided Self-Supervision

2026-03-14 · Yunhe Gao, Yabin Zhang, Chong Wang, Jiaming Liu 외 arxiv

Foundation models have transformed vision and language by learning general-purpose representations from large-scale unlabeled data, yet 3D medical imaging lacks analogous approaches. Existing self-supervised methods rely…

Self-Supervised Learning

DECAR: Deep Clustering for learning general-purpose Audio Representations

2021-10-17 · Sreyan Ghosh, Sandesh V Katta, Ashish Seth, S. Umesh

We introduce DECAR, a self-supervised pre-training approach for learning general-purpose audio representations. Our system is based on clustering: it utilizes an offline clustering step to provide target labels that act …

Clustering Deep Clustering Self-Supervised Learning Transfer Learning

Boosting Supervision with Self-Supervision for Few-shot Learning

2019-06-17 · Jong-Chyi Su, Subhransu Maji, Bharath Hariharan

We present a technique to improve the transferability of deep representations learned on small labeled datasets by introducing self-supervised tasks as auxiliary loss functions. While recent approaches for self-supervise…

Few-Shot Learning Self-Supervised Learning