Revisiting Softmax Masking: Stop Gradient for Enhancing Stability in Replay-based Continual Learning
In replay-based methods for continual learning, replaying input samples in episodic memory has shown its effectiveness in alleviating catastrophic forgetting. However, the potential key factor of cross-entropy loss with softmax in causing catastrophic forgetting has been underexplored. In this paper, we analyze the effect of softmax and revisit softmax masking with negative infinity to shed light on its ability to mitigate catastrophic forgetting. Based on the analyses, it is found that negative infinity masked softmax is not always compatible with dark knowledge. To improve the compatibility, we propose a general masked softmax that controls the stability by adjusting the gradient scale to old and new classes. We demonstrate that utilizing our method on other replay-based methods results in better performance, primarily by enhancing model stability in continual learning benchmarks, even when the buffer size is set to an extremely small value.
Code (0)
등록된 구현이 없습니다.
Tasks
Continual LearningIncremental LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Overcoming Valid Action Suppression in Unmasked Policy Gradient Algorithms
In reinforcement learning environments with state-dependent action validity, action masking consistently outperforms penalty-based handling of invalid actions, yet existing theory only shows that masking preserves the po…
Reinforcement LearningBeyond Stationarity: Convergence Analysis of Stochastic Softmax Policy Gradient Methods
Markov Decision Processes (MDPs) are a formal framework for modeling and solving sequential decision-making problems. In finite-time horizons such problems are relevant for instance for optimal stopping or specific suppl…
Decision MakingPolicy Gradient MethodsSequential Decision MakingRevisiting the Gumbel-Softmax in MADDPG
MADDPG is an algorithm in multi-agent reinforcement learning (MARL) that extends the popular single-agent method, DDPG, to multi-agent scenarios. Importantly, DDPG is an algorithm designed for continuous action spaces, w…
BenchmarkingMulti-agent Reinforcement LearningRevisiting Role of Autoencoders in Adversarial Settings
To combat against adversarial attacks, autoencoder structure is widely used to perform denoising which is regarded as gradient masking. In this paper, we revisit the role of autoencoders in adversarial settings. Through …
Adversarial DefenseAdversarial RobustnessDenoisingExtreme Classification via Adversarial Softmax Approximation
Training a classifier over a large number of classes, known as 'extreme classification', has become a topic of major interest with applications in technology, science, and e-commerce. Traditional softmax regression induc…
ClassificationGeneral Classification