Learn or Recall? Revisiting Incremental Learning with Pre-trained Language Models
Incremental Learning (IL) has been a long-standing problem in both vision and Natural Language Processing (NLP) communities. In recent years, as Pre-trained Language Models (PLMs) have achieved remarkable progress in various NLP downstream tasks, utilizing PLMs as backbones has become a common practice in recent research of IL in NLP. Most assume that catastrophic forgetting is the biggest obstacle to achieving superior IL performance and propose various techniques to overcome this issue. However, we find that this assumption is problematic. Specifically, we revisit more than 20 methods on four classification tasks (Text Classification, Intent Classification, Relation Extraction, and Named Entity Recognition) under the two most popular IL settings (Class-Incremental and Task-Incremental) and reveal that most of them severely underestimate the inherent anti-forgetting ability of PLMs. Based on the observation, we propose a frustratingly easy method called SEQ* for IL with PLMs. The results show that SEQ* has competitive or superior performance compared to state-of-the-art (SOTA) IL methods and requires considerably less trainable parameters and training time. These findings urge us to revisit the IL with PLMs and encourage future studies to have a fundamental understanding of the catastrophic forgetting in PLMs. The data, code and scripts are publicly available at https://github.com/zzz47zzz/codebase-for-incremental-learning-with-llm.
Code (2)
Tasks
Class Incremental LearningIncremental Learningintent-classificationIntent Classificationnamed-entity-recognitionNamed Entity RecognitionRelation Extractiontext-classificationText ClassificationSimilar Papers 제목 키워드 기반
Static-Dynamic Co-Teaching for Class-Incremental 3D Object Detection
Deep learning-based approaches have shown remarkable performance in the 3D object detection task. However, they suffer from a catastrophic performance drop on the originally trained classes when incrementally learning ne…
3D Object DetectionIncremental LearningObjectobject-detection+1Revisiting Class-Incremental Learning with Pre-Trained Models: Generalizability and Adaptivity are All You Need
Class-incremental learning (CIL) aims to adapt to emerging new classes without forgetting old ones. Traditional CIL models are trained from scratch to continually acquire knowledge as data evolves. Recently, pre-training…
Allclass-incremental learningClass Incremental LearningIncremental Learning+1RECALL: Replay-based Continual Learning in Semantic Segmentation
Deep networks allow to obtain outstanding results in semantic segmentation, however they need to be trained in a single shot with a large amount of data. Continual learning settings where new classes are learned in incre…
Continual LearningGenerative Adversarial NetworkSemantic SegmentationIncremental Concept Learning via Online Generative Memory Recall
The ability to learn more and more concepts over time from incrementally arriving data is essential for the development of a life-long learning system. However, deep neural networks often suffer from forgetting previousl…
class-incremental learningClass Incremental LearningGenerative Adversarial NetworkIncremental LearningRevisiting Modularized Multilingual NMT to Meet Industrial Demands
The complete sharing of parameters for multilingual translation (1-1) has been the mainstream approach in current research. However, degraded performance due to the capacity bottleneck and low maintainability hinders its…
Machine TranslationNMTTranslation