Focus on the present: a regularization method for the ASR source-target attention layer
This paper introduces a novel method to diagnose the source-target attention in state-of-the-art end-to-end speech recognition models with joint connectionist temporal classification (CTC) and attention training. Our method is based on the fact that both, CTC and source-target attention, are acting on the same encoder representations. To understand the functionality of the attention, CTC is applied to compute the token posteriors given the attention outputs. We found that the source-target attention heads are able to predict several tokens ahead of the current one. Inspired by the observation, a new regularization method is proposed which leverages CTC to make source-target attention more focused on the frames corresponding to the output token being predicted by the decoder. Experiments reveal stable improvements up to 7\% and 13\% relatively with the proposed regularization on TED-LIUM 2 and LibriSpeech.
Code (0)
등록된 구현이 없습니다.
Tasks
Decoderspeech-recognitionSpeech RecognitionSimilar Papers 제목 키워드 기반
In-context Region-based Drag: Drag Any Region to Any Shape
Diffusion models have shown promise in drag-style editing. Previous works mainly focus on point-based drag, which is inherently ambiguous. This paper focuses on region-based drag and introduces a novel In-Context Region-…
Unsupervised Domain Adaptive Fundus Image Segmentation with Category-level Regularization
Existing unsupervised domain adaptation methods based on adversarial learning have achieved good performance in several medical imaging tasks. However, these methods focus only on global distribution adaptation and ignor…
Domain AdaptationImage SegmentationSemantic SegmentationUnsupervised Domain AdaptationGTA: Guided Transfer of Spatial Attention from Object-Centric Representations
Utilizing well-trained representations in transfer learning often results in superior performance and faster convergence compared to training from scratch. However, even if such good representations are transferred, a mo…
Inductive BiasObject LocalizationTransfer LearningSimulate, Refocus and Ensemble: An Attention-Refocusing Scheme for Domain Generalization
Domain generalization (DG) aims to learn a model from source domains and apply it to unseen target domains with out-of-distribution data. Owing to CLIP's strong ability to encode semantic concepts, it has attracted incre…
Domain GeneralizationEnsemble LearningLautum Regularization for Semi-supervised Transfer Learning
Transfer learning is a very important tool in deep learning as it allows propagating information from one "source dataset" to another "target dataset", especially in the case of a small number of training examples in the…
Transfer Learning