Guiding Frame-Level CTC Alignments Using Self-knowledge Distillation
Transformer encoder with connectionist temporal classification (CTC) framework is widely used for automatic speech recognition (ASR). However, knowledge distillation (KD) for ASR displays a problem of disagreement between teacher-student models in frame-level alignment which ultimately hinders it from improving the student model's performance. In order to resolve this problem, this paper introduces a self-knowledge distillation (SKD) method that guides the frame-level alignment during the training time. In contrast to the conventional method using separate teacher and student models, this study introduces a simple and effective method sharing encoder layers and applying the sub-model as the student model. Overall, our approach is effective in improving both the resource efficiency as well as performance. We also conducted an experimental analysis of the spike timings to illustrate that the proposed method improves performance by reducing the alignment disagreement.
Code (1)
Tasks
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Knowledge DistillationSelf-Knowledge Distillationspeech-recognitionSpeech RecognitionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Guiding CTC Posterior Spike Timings for Improved Posterior Fusion and Knowledge Distillation
Conventional automatic speech recognition (ASR) systems trained from frame-level alignments can easily leverage posterior fusion to improve ASR accuracy and build a better single model with knowledge distillation. End-to…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Data AugmentationDecoder+5SKILL-RAG: Self-Knowledge Induced Learning and Filtering for Retrieval-Augmented Generation
Retrieval-Augmented Generation (RAG) has significantly improved the performance of large language models (LLMs) on knowledge-intensive tasks in recent years. However, since retrieval systems may return irrelevant content…
Reinforcement LearningQuestion AnsweringSARA: Controllable Makeup Transfer with Spatial Alignment and Region-Adaptive Normalization
Makeup transfer is a process of transferring the makeup style from a reference image to the source images, while preserving the source images' identities. This technique is highly desirable and finds many applications. H…
SSPA: Split-and-Synthesize Prompting with Gated Alignments for Multi-Label Image Recognition
Multi-label image recognition is a fundamental task in computer vision. Recently, Vision-Language Models (VLMs) have made notable advancements in this area. However, previous methods fail to effectively leverage the rich…
In-Context LearningMulti-Label Image RecognitionDREEAM: Guiding Attention with Evidence for Improving Document-Level Relation Extraction
Document-level relation extraction (DocRE) is the task of identifying all relations between each entity pair in a document. Evidence, defined as sentences containing clues for the relationship between an entity pair, has…
Document-level Relation ExtractionRelationRelation Extraction