Finding Task-specific Subnetworks in Multi-task Spoken Language Understanding Model
Recently, multi-task spoken language understanding (SLU) models have emerged, designed to address various speech processing tasks. However, these models often rely on a large number of parameters. Also, they often encounter difficulties in adapting to new data for a specific task without experiencing catastrophic forgetting of previously trained tasks. In this study, we propose finding task-specific subnetworks within a multi-task SLU model via neural network pruning. In addition to model compression, we expect that the forgetting of previously trained tasks can be mitigated by updating only a task-specific subnetwork. We conduct experiments on top of the state-of-the-art multi-task SLU model ``UniverSLU'', trained for several tasks such as emotion recognition (ER), intent classification (IC), and automatic speech recognition (ASR). We show that pruned models were successful in adapting to additional ASR or IC data with minimal performance degradation on previously trained tasks.
Code (0)
등록된 구현이 없습니다.
Tasks
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Emotion Recognitionintent-classificationIntent ClassificationModel CompressionNetwork Pruningspeech-recognitionSpeech RecognitionSpoken Language UnderstandingSimilar Papers 제목 키워드 기반
The Quest for Winning Tickets in Low-Rank Adapters
The Lottery Ticket Hypothesis (LTH) suggests that over-parameterized neural networks contain sparse subnetworks ("winning tickets") capable of matching full model performance when trained from scratch. With the growing r…
parameter-efficient fine-tuningTransfer LearningLow-Complexity Probing via Finding Subnetworks
The dominant approach in probing neural networks for linguistic properties is to train a new shallow multi-layer perceptron (MLP) on top of the model's internal representations. This approach can detect properties encode…
Forget-free Continual Learning with Winning Subnetworks
Inspired by Lottery Ticket Hypothesis that competitive subnetworks exist within a dense network, we propose a continual learning method referred to as Winning SubNetworks (WSN), which sequentially learns and selects an o…
Continual LearningPlaying Lottery Tickets with Vision and Language
Large-scale pre-training has recently revolutionized vision-and-language (VL) research. Models such as LXMERT and UNITER have significantly lifted the state of the art over a wide range of VL tasks. However, the large nu…
Image-text RetrievalQuestion AnsweringReferring ExpressionReferring Expression Comprehension+6Learning to Win Lottery Tickets in BERT Transfer via Task-agnostic Mask Training
Recent studies on the lottery ticket hypothesis (LTH) show that pre-trained language models (PLMs) like BERT contain matching subnetworks that have similar transfer learning performance as the original PLM. These subnetw…
Transfer Learning