Continual Learning for Text Classification with Information Disentanglement Based Regularization
Continual learning has become increasingly important as it enables NLP models to constantly learn and gain knowledge over time. Previous continual learning methods are mainly designed to preserve knowledge from previous tasks, without much emphasis on how to well generalize models to new tasks. In this work, we propose an information disentanglement based regularization method for continual learning on text classification. Our proposed method first disentangles text hidden spaces into representations that are generic to all tasks and representations specific to each individual task, and further regularizes these representations differently to better constrain the knowledge required to generalize. We also introduce two simple auxiliary tasks: next sentence prediction and task-id prediction, for learning better generic and specific representation spaces. Experiments conducted on large-scale benchmarks demonstrate the effectiveness of our method in continual text classification tasks with various sequences and lengths over state-of-the-art baselines. We have publicly released our code at https://github.com/GT-SALT/IDBR.
Code (1)
Tasks
Continual LearningDisentanglementGeneral ClassificationSentencetext-classificationText ClassificationSimilar Papers 제목 키워드 기반
Infinite dSprites for Disentangled Continual Learning: Separating Memory Edits from Generalization
The ability of machine learning systems to learn continually is hindered by catastrophic forgetting, the tendency of neural networks to overwrite previously acquired knowledge when learning a new task. Existing methods m…
ClassificationContinual LearningDisentanglementMemorization+1Disentanglement of Color and Shape Representations for Continual Learning
We hypothesize that disentangled feature representations suffer less from catastrophic forgetting. As a case study we perform explicit disentanglement of color and shape, by adjusting the network architecture. We tested …
Continual LearningDisentanglementConDiSR: Contrastive Disentanglement and Style Regularization for Single Domain Generalization
Medical data often exhibits distribution shifts, which cause test-time performance degradation for deep learning models trained using standard supervised learning pipelines. This challenge is addressed in the field of Do…
ClassificationDisentanglementDomain Generalizationimage-classification+2Text Classification and Clustering with Annealing Soft Nearest Neighbor Loss
We define disentanglement as how far class-different data points from each other are, relative to the distances among class-similar data points. When maximizing disentanglement during representation learning, we obtain a…
ClassificationClusteringDisentanglementRepresentation Learning+3Continual Distributed Learning for Crisis Management
Social media platforms such as Twitter, Facebook etc can be utilised as an important source of information during disaster events. This information can be used for disaster response and crisis management if processed acc…
Continual LearningDisaster ResponseFederated LearningManagement+2