Lifelong Language Knowledge Distillation
It is challenging to perform lifelong language learning (LLL) on a stream of different tasks without any performance degradation comparing to the multi-task counterparts. To address this issue, we present Lifelong Language Knowledge Distillation (L2KD), a simple but efficient method that can be easily applied to existing LLL architectures in order to mitigate the degradation. Specifically, when the LLL model is trained on a new task, we assign a teacher model to first learn the new task, and pass the knowledge to the LLL model via knowledge distillation. Therefore, the LLL model can better adapt to the new task while keeping the previously learned knowledge. Experiments show that the proposed L2KD consistently improves previous state-of-the-art models, and the degradation comparing to multi-task models in LLL tasks is well mitigated for both sequence generation and text classification tasks.
Code (1)
Tasks
Knowledge DistillationLanguage Modellingtext-classificationText ClassificationText GenerationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Lifelong Learning via Progressive Distillation and Retrospection
Lifelong learning aims at adapting a learned model to new tasks while retaining the knowledge gained earlier. A key challenge for lifelong learning is how to strike a balance between the preservation on old tasks and the…
Knowledge DistillationLifelong learningA Deep Hierarchical Approach to Lifelong Learning in Minecraft
We propose a lifelong learning system that has the ability to reuse and transfer knowledge from one task to another while efficiently retaining the previously learned knowledge-base. Knowledge is transferred by learning …
Deep Reinforcement LearningLifelong learningMinecraftReinforcement LearningHAD: Heterogeneity-Aware Distillation for Lifelong Heterogeneous Learning
Lifelong learning aims to preserve knowledge acquired from previous tasks while incorporating knowledge from a sequence of new tasks. However, most prior work explores only streams of homogeneous tasks (\textit{e.g.}, on…
Patch-based Knowledge Distillation for Lifelong Person Re-Identification
The task of lifelong person re-identification aims to match a person across multiple cameras given continuous data streams. Similar to other lifelong learning tasks, it severely suffers from the so-called catastrophic fo…
Continual LearningKnowledge DistillationLifelong learningPerson Re-IdentificationLifelong Twin Generative Adversarial Networks
In this paper, we propose a new continuously learning generative model, called the Lifelong Twin Generative Adversarial Networks (LT-GANs). LT-GANs learns a sequence of tasks from several databases and its architecture c…
Knowledge Distillation