Self-Supervised Knowledge Transfer via Loosely Supervised Auxiliary Tasks
Knowledge transfer using convolutional neural networks (CNNs) can help efficiently train a CNN with fewer parameters or maximize the generalization performance under limited supervision. To enable a more efficient transfer of pretrained knowledge under relaxed conditions, we propose a simple yet powerful knowledge transfer methodology without any restrictions regarding the network structure or dataset used, namely self-supervised knowledge transfer (SSKT), via loosely supervised auxiliary tasks. For this, we devise a training methodology that transfers previously learned knowledge to the current training process as an auxiliary task for the target task through self-supervision using a soft label. The SSKT is independent of the network structure and dataset, and is trained differently from existing knowledge transfer methods; hence, it has an advantage in that the prior knowledge acquired from various tasks can be naturally transferred during the training process to the target task. Furthermore, it can improve the generalization performance on most datasets through the proposed knowledge transfer between different problem domains from multiple source networks. SSKT outperforms the other transfer learning methods (KD, DML, and MAXL) through experiments under various knowledge transfer settings. The source code will be made available to the public.
Code (1)
Tasks
Transfer LearningSimilar Papers 제목 키워드 기반
Choice of training label matters: how to best use deep learning for quantitative MRI parameter estimation
Deep learning (DL) is gaining popularity as a parameter estimation method for quantitative MRI. A range of competing implementations have been proposed, relying on either supervised or self-supervised learning. Self-supe…
parameter estimationQuantitative MRISelf-Supervised LearningToward Improved Generalization: Meta Transfer of Self-supervised Knowledge on Graphs
Despite the remarkable success achieved by graph convolutional networks for functional brain activity analysis, the heterogeneity of functional patterns and the scarcity of imaging data still pose challenges in many task…
Meta-LearningRepresentation LearningSelf-Supervised LearningTransfer LearningFunctional Knowledge Transfer with Self-supervised Representation Learning
This work investigates the unexplored usability of self-supervised representation learning in the direction of functional knowledge transfer. In this work, functional knowledge transfer is achieved by joint optimization …
Representation LearningSelf-Supervised LearningTransfer LearningDistilling Knowledge from Self-Supervised Teacher by Embedding Graph Alignment
Recent advances have indicated the strengths of self-supervised pre-training for improving representation learning on downstream tasks. Existing works often utilize self-supervised pre-trained models by fine-tuning on do…
Knowledge DistillationRepresentation LearningSelf-Supervised LearningHierarchical Self-supervised Augmented Knowledge Distillation
Knowledge distillation often involves how to define and transfer knowledge from teacher to student effectively. Although recent self-supervised contrastive knowledge achieves the best performance, forcing the network to …
Knowledge DistillationRepresentation Learning