Exploring Continuous Integrate-and-Fire for Adaptive Simultaneous Speech Translation
Simultaneous speech translation (SimulST) is a challenging task aiming to translate streaming speech before the complete input is observed. A SimulST system generally includes two components: the pre-decision that aggregates the speech information and the policy that decides to read or write. While recent works had proposed various strategies to improve the pre-decision, they mainly adopt the fixed wait-k policy, leaving the adaptive policies rarely explored. This paper proposes to model the adaptive policy by adapting the Continuous Integrate-and-Fire (CIF). Compared with monotonic multihead attention (MMA), our method has the advantage of simpler computation, superior quality at low latency, and better generalization to long utterances. We conduct experiments on the MuST-C V2 dataset and show the effectiveness of our approach.
Code (1)
Tasks
TranslationSimilar Papers 제목 키워드 기반
VILAS: Exploring the Effects of Vision and Language Context in Automatic Speech Recognition
Enhancing automatic speech recognition (ASR) performance by leveraging additional multimodal information has shown promising results in previous studies. However, most of these works have primarily focused on utilizing v…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)speech-recognitionSpeech RecognitionAn adaptive large neighborhood search heuristic for the multi-port continuous berth allocation problem
In this paper, we study a problem that integrates the vessel scheduling problem with the berth allocation into a collaborative problem denoted as the multi-port continuous berth allocation problem (MCBAP). This problem o…
SchedulingCIF: Continuous Integrate-and-Fire for End-to-End Speech Recognition
In this paper, we propose a novel soft and monotonic alignment mechanism used for sequence transduction. It is inspired by the integrate-and-fire model in spiking neural networks and employed in the encoder-decoder frame…
DecoderLanguage ModellingMulti-Task Learningspeech-recognition+1Domain Wall Magnetic Tunnel Junction Reliable Integrate and Fire Neuron
In spiking neural networks, neuron dynamics are described by the biologically realistic integrate-and-fire model that captures membrane potential accumulation and above-threshold firing behaviors. Among the hardware impl…
image-classificationImage ClassificationCIF-PT: Bridging Speech and Text Representations for Spoken Language Understanding via Continuous Integrate-and-Fire Pre-Training
Speech or text representation generated by pre-trained models contains modal-specific information that could be combined for benefiting spoken language understanding (SLU) tasks. In this work, we propose a novel pre-trai…
intent-classificationIntent ClassificationIntent Classification and Slot FillingLanguage Modeling+5