Less-forgetting Learning in Deep Neural Networks
A catastrophic forgetting problem makes deep neural networks forget the previously learned information, when learning data collected in new environments, such as by different sensors or in different light conditions. This paper presents a new method for alleviating the catastrophic forgetting problem. Unlike previous research, our method does not use any information from the source domain. Surprisingly, our method is very effective to forget less of the information in the source domain, and we show the effectiveness of our method using several experiments. Furthermore, we observed that the forgetting problem occurs between mini-batches when performing general training processes using stochastic gradient descent methods, and this problem is one of the factors that degrades generalization performance of the network. We also try to solve this problem using the proposed method. Finally, we show our less-forgetting learning method is also helpful to improve the performance of deep neural networks in terms of recognition rates.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Improving Gender Fairness of Pre-Trained Language Models without Catastrophic Forgetting
Existing studies addressing gender bias of pre-trained language models, usually build a small gender-neutral data set and conduct a second phase pre-training on the model with such data. However, given the limited size a…
coreference-resolutionCoreference ResolutionFairnessForgetting in Language Models: Capacity, Optimization, and Self-Generated Replay
Models trained on a new task typically degrade on prior tasks, a phenomenon known as forgetting. Traditionally, mitigating forgetting has required replaying stored exemplars from prior tasks, which is often impractical. …
Does an LSTM forget more than a CNN? An empirical study of catastrophic forgetting in NLP
Catastrophic forgetting {---} whereby a model trained on one task is fine-tuned on a second, and in doing so, suffers a {``}catastrophic{''} drop in performance over the first task {---} is a hurdle in the development of…
Continual LearningTransfer LearningMore Classifiers, Less Forgetting: A Generic Multi-classifier Paradigm for Incremental Learning
Less Forgetting: A Generic Multi-classifier Paradigm for Incremental Learning","Overcoming catastrophic forgetting in neural networks is a long-standing and core research objective for incremental learning. Notable studi…
Incremental LearningChained Tuning Leads to Biased Forgetting
Large language models (LLMs) are often fine-tuned for use on downstream tasks, though this can degrade capabilities learned during previous training. This phenomenon, often referred to as catastrophic forgetting, has imp…
Continual Learning