A comprehensive, application-oriented study of catastrophic forgetting in DNNs
We present a large-scale empirical study of catastrophic forgetting (CF) in modern Deep Neural Network (DNN) models that perform sequential (or: incremental) learning. A new experimental protocol is proposed that enforces typical constraints encountered in application scenarios. As the investigation is empirical, we evaluate CF behavior on the hitherto largest number of visual classification datasets, from each of which we construct a representative number of Sequential Learning Tasks (SLTs) in close alignment to previous works on CF. Our results clearly indicate that there is no model that avoids CF for all investigated datasets and SLTs under application conditions. We conclude with a discussion of potential solutions and workarounds to CF, notably for the EWC and IMM models.
Code (1)
Tasks
General ClassificationIncremental LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Continual Learning for Natural Language Generation in Task-oriented Dialog Systems
Natural language generation (NLG) is an essential component of task-oriented dialog systems. Despite the recent success of neural approaches for NLG, they are typically developed in an offline manner for particular domai…
Continual LearningText GenerationCatastrophic forgetting: still a problem for DNNs
We investigate the performance of DNNs when trained on class-incremental visual problems consisting of initial training, followed by retraining with added visual classes. Catastrophic forgetting (CF) behavior is measured…
Incremental LearningModel SelectionA Methodology-Oriented Study of Catastrophic Forgetting in Incremental Deep Neural Networks
Human being and different species of animals having the skills to gather, transferring knowledge, processing, fine-tune and generating information throughout their lifetime. The ability of learning throughout their lifes…
Incremental LearningOvercoming Catastrophic Forgetting by Exemplar Selection in Task-oriented Dialogue System
Intelligent task-oriented dialogue systems (ToDs) are expected to continuously acquire new knowledge, also known as Continual Learning (CL), which is crucial to fit ever-changing user needs. However, catastrophic forgett…
Continual LearningTask-Oriented Dialogue SystemsUFO: A Unified Flow-Oriented Framework for Robust Continual Graph Learning
Graph learning research has increasingly shifted toward continual graph learning (CGL), which better reflects real-world scenarios where graphs evolve over time. However, existing CGL methods largely assume clean supervi…
Graph Learning