Exploring the Stability Gap in Continual Learning: The Role of the Classification Head

2024-11-06 · Wojciech Łapacz, Daniel Marczak, Filip Szatkowski, Tomasz Trzciński

Continual learning (CL) has emerged as a critical area in machine learning, enabling neural networks to learn from evolving data distributions while mitigating catastrophic forgetting. However, recent research has identified the stability gap -- a phenomenon where models initially lose performance on previously learned tasks before partially recovering during training. Such learning dynamics are contradictory to the intuitive understanding of stability in continual learning where one would expect the performance to degrade gradually instead of rapidly decreasing and then partially recovering later. To better understand and alleviate the stability gap, we investigate it at different levels of the neural network architecture, particularly focusing on the role of the classification head. We introduce the nearest-mean classifier (NMC) as a tool to attribute the influence of the backbone and the classification head on the stability gap. Our experiments demonstrate that NMC not only improves final performance, but also significantly enhances training stability across various continual learning benchmarks, including CIFAR100, ImageNet100, CUB-200, and FGVC Aircrafts. Moreover, we find that NMC also reduces task-recency bias. Our analysis provides new insights into the stability gap and suggests that the primary contributor to this phenomenon is the linear head, rather than the insufficient representation learning.

📄 PDF Abstract BibTeX arXiv:2411.04723

Code (1)

wojciechl02/stability-gap-nmc 공식 구현 pytorch

Tasks

AttributeContinual Learning Representation Learning

Balancing Stability and Plasticity through Advanced Null Space in Continual Learning

2022-07-25 · Yajing Kong, Liu Liu, Zhen Wang, DaCheng Tao

Continual learning is a learning paradigm that learns tasks sequentially with resources constraints, in which the key challenge is stability-plasticity dilemma, i.e., it is uneasy to simultaneously have the stability to …

Continual Learning

Branch-Tuning: Balancing Stability and Plasticity for Continual Self-Supervised Learning

2024-03-27 · Wenzhuo LIU, Fei Zhu, Cheng-Lin Liu

Self-supervised learning (SSL) has emerged as an effective paradigm for deriving general representations from vast amounts of unlabeled data. However, as real-world applications continually integrate new content, the hig…

Continual Learning Continual Self-Supervised Learning Self-Supervised Learning

Enhancing Pretrained Model-based Continual Representation Learning via Guided Random Projection

2026-03-19 · Ruilin Li, Heming Zou, Xiufeng Yan, Zheming Liang 외 arxiv

Recent paradigms in Random Projection Layer (RPL)-based continual representation learning have demonstrated superior performance when building upon a pre-trained model (PTM). These methods insert a randomly initialized R…

Class Incremental Learning Representation Learning Continual Learning

GCR: Geometry-Consistent Routing for Task-Agnostic Continual Anomaly Detection

2026-01-05 · Joongwon Chae, Lihui Luo, Yang Liu, Runming Wang 외 arxiv

Feature-based anomaly detection is widely adopted in industrial inspection due to the strong representational power of large pre-trained vision encoders. While most existing methods focus on improving within-category ano…

Representation Learning Anomaly Detection Decision Making

Adapter-Based Few-Shot Continual Learning for Malicious Packet Recognition

2026-08-24 · Kyle Stein, Guillermo Francia, III Eman El-Sheikh, Andrew Arash Mahyari arxiv

The continual evolution of malware variants necessitates detection systems that can adapt to new threats without retraining from scratch. However, continually updating models on new data often leads to catastrophic forge…

Few-Shot Class-Incremental Learning Self-Supervised Learning Malware Classification Continual Learning