Continual Distillation Learning: Knowledge Distillation in Prompt-based Continual Learning

2024-07-18 · Qifan Zhang, Yunhui Guo, Yu Xiang

We introduce the problem of continual distillation learning (CDL) in order to use knowledge distillation (KD) to improve prompt-based continual learning (CL) models. The CDL problem is valuable to study since the use of a larger vision transformer (ViT) leads to better performance in prompt-based continual learning. The distillation of knowledge from a large ViT to a small ViT improves the inference efficiency for prompt-based CL models. We empirically found that existing KD methods such as logit distillation and feature distillation cannot effectively improve the student model in the CDL setup. To address this issue, we introduce a novel method named Knowledge Distillation based on Prompts (KDP), in which globally accessible prompts specifically designed for knowledge distillation are inserted into the frozen ViT backbone of the student model. We demonstrate that our KDP method effectively enhances the distillation performance in comparison to existing KD methods in the CDL setup.

📄 PDF Abstract BibTeX arXiv:2407.13911

Code (0)

등록된 구현이 없습니다.

Tasks

Continual Learning Knowledge Distillation

Methods 이 논문이 사용한 방법론

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Attention 설명 없음

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Multi-Head Attention 설명 없음

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

Residual Connection 설명 없음

Vision Transformer The Vision Transformer, or ViT, is a model for image classification that employs a Transformer-like architecture over…

A Closer Look at Rehearsal-Free Continual Learning

2022-03-31 · James Seale Smith, Junjiao Tian, Shaunak Halbe, Yen-Chang Hsu 외

Continual learning is a setting where machine learning models learn novel concepts from continuously shifting training data, while simultaneously avoiding degradation of knowledge on previously seen classes which may dis…

Continual Learning Knowledge Distillation L2 Regularization Novel Concepts

ECLIPSE: Efficient Continual Learning in Panoptic Segmentation with Visual Prompt Tuning

2024-03-29 · CVPR 2024 1 · Beomyoung Kim, Joonsang Yu, Sung Ju Hwang

Panoptic segmentation, combining semantic and instance segmentation, stands as a cutting-edge computer vision task. Despite recent progress with deep learning models, the dynamic nature of real-world applications necessi…

Continual Learning Continual Panoptic Segmentation Instance Segmentation Knowledge Distillation+4

Prompt Tuning for Few-Shot Continual Learning Named Entity Recognition

2025-08-10 · Zhe Ren arxiv

Knowledge distillation has been successfully applied to Continual Learning Named Entity Recognition (CLNER) tasks, by using a teacher model trained on old-class data to distill old-class entities present in new-class dat…

Knowledge Distillation Continual Learning

Provable Contrastive Continual Learning

2024-05-29 · Yichen Wen, Zhiquan Tan, Kaipeng Zheng, Chuanlong Xie 외

Continual learning requires learning incremental tasks with dynamic data distributions. So far, it has been observed that employing a combination of contrastive loss and distillation loss for training in continual learni…

Continual Learning

Unsupervised Continual Clustering via Forward-Backward Knowledge Distillation

2026-06-05 · Mohammadreza Sadeghi, Sareh Soleimani, Zihan Wang, Narges Armanfard arxiv

Unsupervised Continual Learning (UCL) aims to enable neural networks to learn sequential tasks without labels or access to past data. A major challenge in this setting is Catastrophic Forgetting, where models forget prev…