Monotonic Chunkwise Attention
Sequence-to-sequence models with soft attention have been successfully applied to a wide variety of problems, but their decoding process incurs a quadratic time and space cost and is inapplicable to real-time sequence transduction. To address these issues, we propose Monotonic Chunkwise Attention (MoChA), which adaptively splits the input sequence into small chunks over which soft attention is computed. We show that models utilizing MoChA can be trained efficiently with standard backpropagation while allowing online and linear-time decoding at test time. When applied to online speech recognition, we obtain state-of-the-art results and match the performance of a model using an offline soft attention mechanism. In document summarization experiments where we do not expect monotonic alignments, we show significantly improved performance compared to a baseline monotonic attention-based model.
Code (1)
Tasks
Document Summarizationspeech-recognitionSpeech RecognitionSimilar Papers 제목 키워드 기반
Monotonic Chunkwise Attention
Sequence-to-sequence models with soft attention have been successfully applied to a wide variety of problems, but their decoding process incurs a quadratic time and space cost and is inapplicable to real-time sequence tr…
Document Summarizationspeech-recognitionSpeech RecognitionTowards Online End-to-end Transformer Automatic Speech Recognition
The Transformer self-attention network has recently shown promising performance as an alternative to recurrent neural networks in end-to-end (E2E) automatic speech recognition (ASR) systems. However, Transformer has a dr…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Decoderspeech-recognition+1Enhancing Monotonic Multihead Attention for Streaming ASR
We investigate a monotonic multihead attention (MMA) by extending hard monotonic attention to Transformer-based automatic speech recognition (ASR) for online streaming applications. For streaming inference, all monotonic…
AllAutomatic Speech RecognitionAutomatic Speech Recognition (ASR)Boundary Detection+2Multi-head Monotonic Chunkwise Attention For Online Speech Recognition
The attention mechanism of the Listen, Attend and Spell (LAS) model requires the whole input sequence to calculate the attention context and thus is not suitable for online speech recognition. To deal with this problem, …
speech-recognitionSpeech RecognitionCTC-synchronous Training for Monotonic Attention Model
Monotonic chunkwise attention (MoChA) has been studied for the online streaming automatic speech recognition (ASR) based on a sequence-to-sequence framework. In contrast to connectionist temporal classification (CTC), ba…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Decodermodel+2