Mitigating Forgetting in Continual Learning with Selective Gradient Projection

2026-02-08 · Anika Singh, Aayush Dhaulakhandi, Varun Chopade, Likhith Malipati, David Martinez, Kevin Zhu arxiv

As neural networks are increasingly deployed in dynamic environments, they face the challenge of catastrophic forgetting, the tendency to overwrite previously learned knowledge when adapting to new tasks, resulting in severe performance degradation on earlier tasks. We propose Selective Forgetting-Aware Optimization (SFAO), a dynamic method that regulates gradient directions via cosine similarity and per-layer gating, enabling controlled forgetting while balancing plasticity and stability. SFAO selectively projects, accepts, or discards updates using a tunable mechanism with efficient Monte Carlo approximation. Experiments on standard continual learning benchmarks show that SFAO achieves competitive accuracy with markedly lower memory cost, a 90$\%$ reduction, and improved forgetting on MNIST datasets, making it suitable for resource-constrained scenarios.

📄 PDF Abstract BibTeX arXiv:2603.26671

Code (0)

등록된 구현이 없습니다.

Tasks

Continual Learning

Continual Gradient Low-Rank Projection Fine-Tuning for LLMs

2025-07-03 · Chenxu Wang, Yilin Lyu, Zicheng Sun, Liping Jing

Continual fine-tuning of Large Language Models (LLMs) is hampered by the trade-off between efficiency and expressiveness. Low-Rank Adaptation (LoRA) offers efficiency but constrains the model's ability to learn new tasks…

Continual Learning

GradMix: Gradient-based Selective Mixup for Robust Data Augmentation in Class-Incremental Learning

2025-05-13 · Minsu Kim, Seong-Hyeon Hwang, Steven Euijong Whang

In the context of continual learning, acquiring new knowledge while maintaining previous knowledge presents a significant challenge. Existing methods often use experience replay techniques that store a small portion of p…

class-incremental learning Class Incremental Learning Continual Learning Data Augmentation+1

On the Implicit Adversariality of Catastrophic Forgetting in Deep Continual Learning

2025-10-10 · Ze Peng, Jian Zhang, Jintao Guo, Lei Qi 외 arxiv

Continual learning seeks the human-like ability to accumulate new skills in machine intelligence. Its central challenge is catastrophic forgetting, whose underlying cause has not been fully understood for deep networks. …

Continual Learning Adversarial Attack

BECAME: BayEsian Continual Learning with Adaptive Model MErging

2025-04-03 · Mei Li, Yuxiang Lu, Qinyan Dai, Suizhi Huang 외

Continual Learning (CL) strives to learn incrementally across tasks while mitigating catastrophic forgetting. A key challenge in CL is balancing stability (retaining prior knowledge) and plasticity (learning new tasks). …

Continual Learning model

Class-Prototype Conditional Diffusion Model with Gradient Projection for Continual Learning

2023-12-10 · Khanh Doan, Quyen Tran, Tung Lam Tran, Tuan Nguyen 외

Mitigating catastrophic forgetting is a key hurdle in continual learning. Deep Generative Replay (GR) provides techniques focused on generating samples from prior tasks to enhance the model's memory capabilities using ge…

Continual Learning Denoising Image Generation