Self-Supervised GAN to Counter Forgetting
GANs involve training two networks in an adversarial game, where each network's task depends on its adversary. Recently, several works have framed GAN training as an online or continual learning problem. We focus on the discriminator, which must perform classification under an (adversarially) shifting data distribution. When trained on sequential tasks, neural networks exhibit \emph{forgetting}. For GANs, discriminator forgetting leads to training instability. To counter forgetting, we encourage the discriminator to maintain useful representations by adding a self-supervision. Conditional GANs have a similar effect using labels. However, our self-supervised GAN does not require labels, and closes the performance gap between conditional and unconditional models. We show that, in doing so, the self-supervised discriminator learns better representations than regular GANs.
Code (0)
등록된 구현이 없습니다.
Tasks
Continual LearningGeneral ClassificationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
MiLorE-SSL: Scaling Multilingual Capabilities in Self-Supervised Models without Forgetting
Self-supervised learning (SSL) has greatly advanced speech representation learning, but multilingual SSL models remain constrained to languages encountered during pretraining. Retraining from scratch to incorporate new l…
Self-Supervised LearningRepresentation LearningS3C: Self-Supervised Stochastic Classifiers for Few-Shot Class-Incremental Learning
Few-shot class-incremental learning (FSCIL) aims to learn progressively about new classes with very few labeled samples, without forgetting the knowledge of already learnt classes. FSCIL suffers from two major challenges…
class-incremental learningClass Incremental LearningFew-Shot Class-Incremental LearningIncremental LearningControlling Forgetting with Test-Time Data in Continual Learning
Foundational vision-language models have shown impressive performance on various downstream tasks. Yet, there is still a pressing need to update these models later as new tasks or domains become available. Ongoing Contin…
Continual LearningPreserving Knowledge in Large Language Model with Model-Agnostic Self-Decompression
Humans can retain old knowledge while learning new information, but Large Language Models (LLMs) often suffer from catastrophic forgetting when post-pretrained or supervised fine-tuned (SFT) on domain-specific data. More…
Language ModelingLanguage ModellingLarge Language ModelmodelTalking to Yourself: Defying Forgetting in Large Language Models
Catastrophic forgetting remains a major challenge when fine-tuning large language models (LLMs) on narrow, task-specific data, often degrading their general knowledge and reasoning abilities. We propose SA-SFT, a lightwe…
General Knowledge