Dropout as an Implicit Gating Mechanism For Continual Learning

2020-04-24 · Seyed-Iman Mirzadeh, Mehrdad Farajtabar, Hassan Ghasemzadeh

In recent years, neural networks have demonstrated an outstanding ability to achieve complex learning tasks across various domains. However, they suffer from the "catastrophic forgetting" problem when they face a sequence of learning tasks, where they forget the old ones as they learn new tasks. This problem is also highly related to the "stability-plasticity dilemma". The more plastic the network, the easier it can learn new tasks, but the faster it also forgets previous ones. Conversely, a stable network cannot learn new tasks as fast as a very plastic network. However, it is more reliable to preserve the knowledge it has learned from the previous tasks. Several solutions have been proposed to overcome the forgetting problem by making the neural network parameters more stable, and some of them have mentioned the significance of dropout in continual learning. However, their relationship has not been sufficiently studied yet. In this paper, we investigate this relationship and show that a stable network with dropout learns a gating mechanism such that for different tasks, different paths of the network are active. Our experiments show that the stability achieved by this implicit gating plays a very critical role in leading to performance comparable to or better than other involved continual learning algorithms to overcome catastrophic forgetting.

📄 PDF Abstract BibTeX arXiv:2004.11545

Code (2)

imirzadeh/stable-continual-learning 공식 구현 pytorch

LeeGitaek/Kaggle_Solving

Tasks

Continual Learning

Methods 이 논문이 사용한 방법론

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

TaskDrop: A Competitive Baseline for Continual Learning of Sentiment Classification

2021-11-05 · JianPing Mei, Yilun Zheng, Qianwei Zhou, Rui Yan

In this paper, we study the multi-task sentiment classification problem in the continual learning setting, i.e., a model is sequentially trained to classifier the sentiment of reviews of products in a particular category…

Classification Continual Learning Sentiment Analysis Sentiment Classification

Sparsity and Heterogeneous Dropout for Continual Learning in the Null Space of Neural Activations

2022-03-12 · Ali Abbasi, Parsa Nooralinejad, Vladimir Braverman, Hamed Pirsiavash 외

Continual/lifelong learning from a non-stationary input data stream is a cornerstone of intelligence. Despite their phenomenal performance in a wide variety of applications, deep neural networks are prone to forgetting t…

Continual Learning Lifelong learning

DropoutDAgger: A Bayesian Approach to Safe Imitation Learning

2017-09-18 · Kunal Menda, Katherine Driggs-Campbell, Mykel J. Kochenderfer

While imitation learning is becoming common practice in robotics, this approach often suffers from data mismatch and compounding errors. DAgger is an iterative algorithm that addresses these issues by continually aggrega…

Imitation Learning MuJoCo

Understanding Generalization and Forgetting in In-Context Continual Learning

2026-05-27 · Guangyu Li, Meng Ding, Lijie Hu arxiv

In-context learning (ICL) derives its power from enabling Large Language Models to adapt to new tasks via prompt-based reasoning alone, entirely bypassing the need for parameter updates. Existing theories primarily study…