Student Becoming the Master: Knowledge Amalgamation for Joint Scene Parsing, Depth Estimation, and More
In this paper, we investigate a novel deep-model reusing task. Our goal is to train a lightweight and versatile student model, without human-labelled annotations, that amalgamates the knowledge and masters the expertise of two pretrained teacher models working on heterogeneous problems, one on scene parsing and the other on depth estimation. To this end, we propose an innovative training strategy that learns the parameters of the student intertwined with the teachers, achieved by 'projecting' its amalgamated features onto each teacher's domain and computing the loss. We also introduce two options to generalize the proposed training strategy to handle three or more tasks simultaneously. The proposed scheme yields very encouraging results. As demonstrated on several benchmarks, the trained student model achieves results even superior to those of the teachers in their own expertise domains and on par with the state-of-the-art fully supervised models relying on human-labelled annotations.
Code (1)
Tasks
Depth EstimationScene ParsingSimilar Papers 제목 키워드 기반
Knowledge Amalgamation for Object Detection with Transformers
Knowledge amalgamation (KA) is a novel deep model reusing task aiming to transfer knowledge from several well-trained teachers to a multi-talented and compact student. Currently, most of these approaches are tailored for…
Objectobject-detectionObject DetectionContrastive Knowledge Amalgamation for Unsupervised Image Classification
Knowledge amalgamation (KA) aims to learn a compact student model to handle the joint objective from multiple teacher models that are are specialized for their own tasks respectively. Current methods focus on coarsely al…
Classificationimage-classificationImage ClassificationUnsupervised Image ClassificationSelf-Regulated Data-Free Knowledge Amalgamation for Text Classification
Recently, there has been a growing availability of pre-trained text models on various model repositories. These models greatly reduce the cost of training new models from scratch as they can be fine-tuned for specific ta…
text-classificationText ClassificationTransfer LearningClass-Incremental Learning via Knowledge Amalgamation
Catastrophic forgetting has been a significant problem hindering the deployment of deep learning algorithms in the continual learning setting. Numerous methods have been proposed to address the catastrophic forgetting pr…
class-incremental learningClass Incremental LearningContinual LearningIncremental LearningModel Uncertainty-Aware Knowledge Amalgamation for Pre-Trained Language Models
As many fine-tuned pre-trained language models~(PLMs) with promising performance are generously released, investigating better ways to reuse these models is vital as it can greatly reduce the retraining computational cos…