A Transformer with Interleaved Self-attention and Convolution for Hybrid Acoustic Models
Transformer with self-attention has achieved great success in the area of nature language processing. Recently, there have been a few studies on transformer for end-to-end speech recognition, while its application for hybrid acoustic model is still very limited. In this paper, we revisit the transformer-based hybrid acoustic model, and propose a model structure with interleaved self-attention and 1D convolution, which is proven to have faster convergence and higher recognition accuracy. We also study several aspects of the transformer model, including the impact of the positional encoding feature, dropout regularization, as well as training with and without time restriction. We show competitive recognition results on the public Librispeech dataset when compared to the Kaldi baseline at both cross entropy training and sequence training stages. For reproducible research, we release our source code and recipe within the PyKaldi2 toolbox.
Code (1)
Tasks
speech-recognitionSpeech RecognitionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Conv-Transformer Transducer: Low Latency, Low Frame Rate, Streamable End-to-End Speech Recognition
Transformer has achieved competitive performance against state-of-the-art end-to-end models in automatic speech recognition (ASR), and requires significantly less training time than RNN-based models. The original Transfo…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Decoderspeech-recognition+1nnFormer: Interleaved Transformer for Volumetric Segmentation
Transformer, the model of choice for natural language processing, has drawn scant attention from the medical imaging community. Given the ability to exploit long-term dependencies, transformers are promising to help atyp…
Image SegmentationInductive BiasMedical Image SegmentationSegmentation+2Iwin Transformer: Hierarchical Vision Transformer using Interleaved Windows
We introduce Iwin Transformer, a novel position-embedding-free hierarchical vision transformer, which can be fine-tuned directly from low to high resolution, through the collaboration of innovative interleaved window att…
Conditional Image GenerationSemantic SegmentationImage ClassificationAction RecognitionComponent-Aware Self-Speculative Decoding in Hybrid Language Models
Speculative decoding accelerates autoregressive inference by drafting candidate tokens with a fast model and verifying them in parallel with the target. Self-speculative methods avoid the need for an external drafter but…
MaxViT-UNet: Multi-Axis Attention for Medical Image Segmentation
Since their emergence, Convolutional Neural Networks (CNNs) have made significant strides in medical image analysis. However, the local nature of the convolution operator may pose a limitation for capturing global and lo…
DecoderFlood extent forecastingImage SegmentationInductive Bias+4