The impact of model size on catastrophic forgetting in Online Continual Learning
This study investigates the impact of model size on Online Continual Learning performance, with a focus on catastrophic forgetting. Employing ResNet architectures of varying sizes, the research examines how network depth and width affect model performance in class-incremental learning using the SplitCIFAR-10 dataset. Key findings reveal that larger models do not guarantee better Continual Learning performance; in fact, they often struggle more in adapting to new tasks, particularly in online settings. These results challenge the notion that larger models inherently mitigate catastrophic forgetting, highlighting the nuanced relationship between model size and Continual Learning efficacy. This study contributes to a deeper understanding of model scalability and its practical implications in Continual Learning scenarios.
Code (0)
등록된 구현이 없습니다.
Tasks
class-incremental learningClass Incremental LearningContinual LearningIncremental LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Online Distillation with Continual Learning for Cyclic Domain Shifts
In recent years, online distillation has emerged as a powerful technique for adapting real-time deep neural networks on the fly using a slow, but accurate teacher model. However, a major challenge in online distillation …
Autonomous DrivingContinual LearningOnline Coreset Selection for Rehearsal-based Continual Learning
A dataset is a shred of crucial evidence to describe a task. However, each data point in the dataset does not have the same potential, as some of the data points can be more representative or informative than others. Thi…
Continual LearningDisentanglement of Color and Shape Representations for Continual Learning
We hypothesize that disentangled feature representations suffer less from catastrophic forgetting. As a case study we perform explicit disentanglement of color and shape, by adjusting the network architecture. We tested …
Continual LearningDisentanglementWide Neural Networks Forget Less Catastrophically
A primary focus area in continual learning research is alleviating the "catastrophic forgetting" problem in neural networks by designing new algorithms that are more robust to the distribution shifts. While the recent pr…
Continual LearningContinual Learning with Adaptive Weights (CLAW)
Approaches to continual learning aim to successfully learn a set of related tasks that arrive in an online manner. Recently, several frameworks have been developed which enable deep learning to be deployed in this learni…
Continual LearningTransfer LearningVariational Inference