Continual Learning in Linear Classification on Separable Data
We analyze continual learning on a sequence of separable linear classification tasks with binary labels. We show theoretically that learning with weak regularization reduces to solving a sequential max-margin problem, corresponding to a special case of the Projection Onto Convex Sets (POCS) framework. We then develop upper bounds on the forgetting and other quantities of interest under various settings with recurring tasks, including cyclic and random orderings of tasks. We discuss several practical implications to popular training practices like regularization scheduling and weighting. We point out several theoretical differences between our continual classification setting and a recently studied continual regression setting.
Code (0)
등록된 구현이 없습니다.
Tasks
ClassificationContinual LearningregressionSchedulingSimilar Papers 제목 키워드 기반
Convergence and Implicit Bias of Gradient Descent on Continual Linear Classification
We study continual learning on multiple linear classification tasks by sequentially running gradient descent (GD) for a fixed budget of iterations per task. When all tasks are jointly linearly separable and are presented…
Continual LearningTransfer LearningBetter Rates for Random Task Orderings in Continual Linear Models
We study the common continual learning setup where an overparameterized model is sequentially fitted to a set of jointly realizable tasks. We analyze the forgetting, i.e., loss on previously seen tasks, after $k$ iterati…
Continual LearningExploring Open-world Continual Learning with Knowns-Unknowns Knowledge Transfer
Open-World Continual Learning (OWCL) is a challenging paradigm where models must incrementally learn new knowledge without forgetting while operating under an open-world assumption. This requires handling incomplete trai…
Continual LearningTransfer LearningBandit Multiclass Linear Classification for the Group Linear Separable Case
We consider the online multiclass linear classification under the bandit feedback setting. Beygelzimer, P\'{a}l, Sz\"{o}r\'{e}nyi, Thiruvenkatachari, Wei, and Zhang [ICML'19] considered two notions of linear separability…
ClassificationGeneral ClassificationOne-Class Semi-Supervised Learning: Detecting Linearly Separable Class by its Mean
In this paper, we presented a novel semi-supervised one-class classification algorithm which assumes that class is linearly separable from other elements. We proved theoretically that class is linearly separable if and o…
General ClassificationOne-Class Classification