Pretrained Language Model in Continual Learning: A Comparative Study
Continual learning (CL) is a real-world learning paradigm in which a model learns from a stream of incoming data while avoiding forgetting previously learned knowledge. Pre-trained language models (PLM) have been successfully employed in the continual learning of different natural-language problems. With the rapid development of many CL methods and PLMs, understanding and disentangling their interactions become essential for the continued improvement of CL performance. In this paper, we thoroughly compare the continual learning performance over the combination of 5 PLMs and 4 veins of CL methods on 3 benchmarks in 2 typical incremental settings. As the probing analysis dissects PLM's performance characteristics in a layer-wise and task-wise way, we propose a simple yet effective method ICLL (Introspective Continual Language Learning) which updates the inner connection of the pre-trained model to adapt to continual learning. Our experiments on three incremental sequence classification benchmarks show the generalizability of the proposed methods over different pre-trained language models.
Code (0)
등록된 구현이 없습니다.
Tasks
Continual LearningLanguage ModelingLanguage ModellingSimilar Papers 제목 키워드 기반
The Construction of Instruction-tuned LLMs for Finance without Instruction Data Using Continual Pretraining and Model Merging
This paper proposes a novel method for constructing instruction-tuned large language models (LLMs) for finance without instruction data. Traditionally, developing such domain-specific LLMs has been resource-intensive, re…
Continual PretrainingSimple Recipe Works: Vision-Language-Action Models are Natural Continual Learners with Reinforcement Learning
Continual Reinforcement Learning (CRL) for Vision-Language-Action (VLA) models is a promising direction toward self-improving embodied agents that can adapt in openended, evolving environments. However, conventional wisd…
Zero-shot GeneralizationReinforcement LearningContinual LearningCL-MASR: A Continual Learning Benchmark for Multilingual ASR
Modern multilingual automatic speech recognition (ASR) systems like Whisper have made it possible to transcribe audio in multiple languages with a single model. However, current state-of-the-art ASR models are typically …
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Continual Learningspeech-recognition+1Pretrained Vision-Language-Action Models are Surprisingly Resistant to Forgetting in Continual Learning
Continual learning is a long-standing challenge in robot policy learning, where a policy must acquire new skills over time without catastrophically forgetting previously learned ones. While prior work has extensively stu…
Continual LearningCost of Reasoning in non-English Languages: A Case Study on Japanese
Reasoning Language Models (RLMs) achieve their strongest performance when they reason in English, the language for which reasoning-oriented training data is most abundant. However, reasoning trace is a clue for model int…