Towards Rehearsal-Free Multilingual ASR: A LoRA-based Case Study on Whisper
Pre-trained multilingual speech foundation models, like Whisper, have shown impressive performance across different languages. However, adapting these models to new or specific languages is computationally extensive and faces catastrophic forgetting problems. Addressing these issues, our study investigates strategies to enhance the model on new languages in the absence of original training data, while also preserving the established performance on the original languages. Specifically, we first compare various LoRA-based methods to find out their vulnerability to forgetting. To mitigate this issue, we propose to leverage the LoRA parameters from the original model for approximate orthogonal gradient descent on the new samples. Additionally, we also introduce a learnable rank coefficient to allocate trainable parameters for more efficient training. Our experiments with a Chinese Whisper model (for Uyghur and Tibetan) yield better results with a more compact parameter set.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Online-LoRA: Task-free Online Continual Learning via Low Rank Adaptation
Catastrophic forgetting is a significant challenge in online continual learning (OCL), especially for non-stationary data streams that do not have well-defined task boundaries. This challenge is exacerbated by the memory…
Continual LearningREAL-OW: Rehearsal-free Open World Object Detection with Low-Rank Adaptation and Dual-Stage Objectness Modeling
Open-World Object Detection (OWOD) requires detectors to identify previously unseen objects as unknown and incrementally incorporate them into the set of known categories, while preserving previously acquired knowledge. …
Open World Object DetectionRehearsal-Free Continual Learning over Small Non-I.I.D. Batches
Robotic vision is a field where continual learning can play a significant role. An embodied agent operating in a complex environment subject to frequent and unpredictable changes is required to learn and adapt continuous…
class-incremental learningClass Incremental LearningContinual LearningObject RecognitionRehearsal-free and Task-free Online Continual Learning With Contrastive Prompt
The main challenge of continual learning is \textit{catastrophic forgetting}. Because of processing data in one pass, online continual learning (OCL) is one of the most difficult continual learning scenarios. To address …
Continual LearningCL-LoRA: Continual Low-Rank Adaptation for Rehearsal-Free Class-Incremental Learning
Class-Incremental Learning (CIL) aims to learn new classes sequentially while retaining the knowledge of previously learned classes. Recently, pre-trained models (PTMs) combined with parameter-efficient fine-tuning (PEFT…
class-incremental learningClass Incremental LearningContinual LearningIncremental Learning+2