Learning Equi-angular Representations for Online Continual Learning
Online continual learning suffers from an underfitted solution due to insufficient training for prompt model update (e.g., single-epoch training). To address the challenge, we propose an efficient online continual learning method using the neural collapse phenomenon. In particular, we induce neural collapse to form a simplex equiangular tight frame (ETF) structure in the representation space so that the continuously learned model with a single epoch can better fit to the streamed data by proposing preparatory data training and residual correction in the representation space. With an extensive set of empirical validations using CIFAR-10/100, TinyImageNet, ImageNet-200, and ImageNet-1K, we show that our proposed method outperforms state-of-the-art methods by a noticeable margin in various online continual learning scenarios such as disjoint and Gaussian scheduled continuous (i.e., boundary-free) data setups.
Code (1)
Tasks
Continual LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Learning Representations on the Unit Sphere: Investigating Angular Gaussian and von Mises-Fisher Distributions for Online Continual Learning
We use the maximum a posteriori estimation principle for learning representations distributed on the unit sphere. We propose to use the angular Gaussian distribution, which corresponds to a Gaussian projected on the unit…
Continual LearningRepresentation LearningRandom Representations Outperform Online Continually Learned Representations
Continual learning has primarily focused on the issue of catastrophic forgetting and the associated stability-plasticity tradeoffs. However, little attention has been paid to the efficacy of continually learned represent…
Continual LearningExemplar-FreeRepresentation LearningClassifications of Single-input Lower Triangular Forms
The purposes of this paper are to classify lower triangular forms and to determine under what conditions a nonlinear system is equivalent to a specific type of lower triangular forms. According to the least multi-indices…
FormVocal Bursts Type PredictionFrom Selection to Scheduling: Federated Geometry-Aware Correction Makes Exemplar Replay Work Better under Continual Dynamic Heterogeneity
Exemplar replay has become an effective strategy for mitigating catastrophic forgetting in federated continual learning (FCL) by retaining representative samples from past tasks. Existing studies focus on designing sampl…
Knowledge DistillationContinual LearningOvercoming Domain Drift in Online Continual Learning
Online Continual Learning (OCL) empowers machine learning models to acquire new knowledge online across a sequence of tasks. However, OCL faces a significant challenge: catastrophic forgetting, wherein the model learned …
Continual Learning