AFD-SLU: Adaptive Feature Distillation for Spoken Language Understanding
Spoken Language Understanding (SLU) is a core component of conversational systems, enabling machines to interpret user utterances. Despite its importance, developing effective SLU systems remains challenging due to the scarcity of labeled training data and the computational burden of deploying Large Language Models (LLMs) in real-world applications. To further alleviate these issues, we propose an Adaptive Feature Distillation framework that transfers rich semantic representations from a General Text Embeddings (GTE)-based teacher model to a lightweight student model. Our method introduces a dynamic adapter equipped with a Residual Projection Neural Network (RPNN) to align heterogeneous feature spaces, and a Dynamic Distillation Coefficient (DDC) that adaptively modulates the distillation strength based on real-time feedback from intent and slot prediction performance. Experiments on the Chinese profile-based ProSLU benchmark demonstrate that AFD-SLU achieves state-of-the-art results, with 95.67% intent accuracy, 92.02% slot F1 score, and 85.50% overall accuracy.
Code (0)
등록된 구현이 없습니다.
Tasks
Spoken Language UnderstandingSimilar Papers 제목 키워드 기반
Adaptive Knowledge Distillation between Text and Speech Pre-trained Models
Learning on a massive amount of speech corpus leads to the recent success of many self-supervised speech models. With knowledge distillation, these models may also benefit from the knowledge encoded by language models th…
Knowledge DistillationSpoken Language UnderstandingTwo-stage Textual Knowledge Distillation for End-to-End Spoken Language Understanding
End-to-end approaches open a new way for more accurate and efficient spoken language understanding (SLU) systems by alleviating the drawbacks of traditional pipeline systems. Previous works exploit textual information fo…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Data AugmentationKnowledge Distillation+3I$^2$KD-SLU: An Intra-Inter Knowledge Distillation Framework for Zero-Shot Cross-Lingual Spoken Language Understanding
Spoken language understanding (SLU) typically includes two subtasks: intent detection and slot filling. Currently, it has achieved great success in high-resource languages, but it still remains challenging in low-resourc…
Intent DetectionKnowledge Distillationslot-fillingSlot Filling+1An Investigation of the Combination of Rehearsal and Knowledge Distillation in Continual Learning for Spoken Language Understanding
Continual learning refers to a dynamical framework in which a model receives a stream of non-stationary data over time and must adapt to new data while preserving previously acquired knowledge. Unluckily, neural networks…
class-incremental learningClass Incremental LearningContinual LearningIncremental Learning+2Contrastive Learning for Improving ASR Robustness in Spoken Language Understanding
Spoken language understanding (SLU) is an essential task for machines to understand human speech for better interactions. However, errors from the automatic speech recognizer (ASR) usually hurt the understanding performa…
Contrastive LearningSpoken Language Understanding