A Streaming Approach For Efficient Batched Beam Search
We propose an efficient batching strategy for variable-length decoding on GPU architectures. During decoding, when candidates terminate or are pruned according to heuristics, our streaming approach periodically "refills" the batch before proceeding with a selected subset of candidates. We apply our method to variable-width beam search on a state-of-the-art machine translation model. Our method decreases runtime by up to 71% compared to a fixed-width beam search baseline and 17% compared to a variable-width baseline, while matching baselines' BLEU. Finally, experiments show that our method can speed up decoding in other domains, such as semantic and syntactic parsing.
Code (1)
Tasks
GPUMachine TranslationTranslationSimilar Papers 제목 키워드 기반
TurboBias 2.0: Streaming Context-Biasing for Production-Efficient ASR Systems
Contextualization is essential for production automatic speech recognition (ASR) systems, where user-provided phrases must be recognized accurately under strict latency constraints. Although many context-biasing methods …
Speech RecognitionVAD-free Streaming Hybrid CTC/Attention ASR for Unsegmented Recording
In this work, we propose novel decoding algorithms to enable streaming automatic speech recognition (ASR) on unsegmented long-form recordings without voice activity detection (VAD), based on monotonic chunkwise attention…
Action DetectionActivity DetectionAutomatic Speech RecognitionAutomatic Speech Recognition (ASR)+3Accelerating NMT Batched Beam Decoding with LMBR Posteriors for Deployment
We describe a batched beam decoding algorithm for NMT with LMBR n-gram posteriors, showing that LMBR techniques still yield gains on top of the best recently reported results with Transformers. We also discuss accelerati…
NMTStreaming Reinforcement Learning under Partial Observability with Real-Time Recurrent Learning
Streaming reinforcement learning has emerged as an online learning paradigm that conforms to the restrictions of natural learning agents that process data incrementally, i.e. with a batch size of 1 and no replay buffer. …
Reinforcement LearningContinuous ControlEffective Batching for Recurrent Neural Network Grammars
As a language model that integrates traditional symbolic operations and flexible neural representations, recurrent neural network grammars (RNNGs) have attracted great attention from both scientific and engineering persp…
GPULanguage ModelingLanguage Modelling