How catastrophic can catastrophic forgetting be in linear regression?
To better understand catastrophic forgetting, we study fitting an overparameterized linear model to a sequence of tasks with different input distributions. We analyze how much the model forgets the true labels of earlier tasks after training on subsequent tasks, obtaining exact expressions and bounds. We establish connections between continual learning in the linear setting and two other research areas: alternating projections and the Kaczmarz method. In specific settings, we highlight differences between forgetting and convergence to the offline solution as studied in those areas. In particular, when T tasks in d dimensions are presented cyclically for k iterations, we prove an upper bound of T^2 * min{1/sqrt(k), d/k} on the forgetting. This stands in contrast to the convergence to the offline solution, which can be arbitrarily slow according to existing alternating projection results. We further show that the T^2 factor can be lifted when tasks are presented in a random ordering.
Code (0)
등록된 구현이 없습니다.
Tasks
Continual LearningregressionSimilar Papers 제목 키워드 기반
Neural Linear Bandits: Overcoming Catastrophic Forgetting through Likelihood Matching
We study neural-linear bandits for solving problems where both exploration and representation learning play an important role. Neural-linear bandits leverage the representation power of deep neural networks and combine i…
Efficient ExplorationMulti-Armed BanditsregressionRepresentation Learning+1Deep Neural Linear Bandits: Overcoming Catastrophic Forgetting through Likelihood Matching
We study the neural-linear bandit model for solving sequential decision-making problems with high dimensional side information. Neural-linear bandits leverage the representation power of deep neural networks and combine …
Decision MakingEfficient ExplorationGeneral ClassificationMulti-Armed Bandits+3The Joint Effect of Task Similarity and Overparameterization on Catastrophic Forgetting -- An Analytical Model
In continual learning, catastrophic forgetting is affected by multiple aspects of the tasks. Previous works have analyzed separately how forgetting is affected by either task similarity or overparameterization. In contra…
Continual LearningregressionAnalysis of Overparameterization in Continual Learning under a Linear Model
Autonomous machine learning systems that learn many tasks in sequence are prone to the catastrophic forgetting problem. Mathematical theory is needed in order to understand the extent of forgetting during continual learn…
Continual LearningregressionPMF-CL: Pareto-Minimal-Forgetting Continual Learner for Conflicting Tasks
In the literature, many continual learning (CL) algorithms have been proposed to address the issue of catastrophic forgetting in ML models (i.e., learning new tasks leads to the loss of performance on previously learned …
Multi-Task LearningContinual Learning