Exploring Speech Recognition, Translation, and Understanding with Discrete Speech Units: A Comparative Study
Speech signals, typically sampled at rates in the tens of thousands per second, contain redundancies, evoking inefficiencies in sequence modeling. High-dimensional speech features such as spectrograms are often used as the input for the subsequent model. However, they can still be redundant. Recent investigations proposed the use of discrete speech units derived from self-supervised learning representations, which significantly compresses the size of speech data. Applying various methods, such as de-duplication and subword modeling, can further compress the speech sequence length. Hence, training time is significantly reduced while retaining notable performance. In this study, we undertake a comprehensive and systematic exploration into the application of discrete units within end-to-end speech processing models. Experiments on 12 automatic speech recognition, 3 speech translation, and 1 spoken language understanding corpora demonstrate that discrete units achieve reasonably good results in almost all the settings. We intend to release our configurations and trained models to foster future research efforts.
Code (0)
등록된 구현이 없습니다.
Tasks
Automatic Speech RecognitionSelf-Supervised Learningspeech-recognitionSpeech RecognitionSpoken Language UnderstandingSimilar Papers 제목 키워드 기반
LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPT
Generative Pre-trained Transformer (GPT) models have achieved remarkable performance on various natural language processing tasks, and have shown great potential as backbones for audio-and-text large language models (LLM…
Audio captioningAutomatic Speech RecognitionEmotion RecognitionLanguage Modelling+14Analyzing Speech Unit Selection for Textless Speech-to-Speech Translation
Recent advancements in textless speech-to-speech translation systems have been driven by the adoption of self-supervised learning techniques. Although most state-of-the-art systems adopt a similar architecture to transfo…
Automatic Speech RecognitionEmotion Recognitionfeature selectionResynthesis+7UWSpeech: Speech to Speech Translation for Unwritten Languages
Existing speech to speech translation systems heavily rely on the text of target language: they usually translate source language either to target text and then synthesize target speech from text, or directly to target s…
speech-recognitionSpeech RecognitionSpeech-to-Speech TranslationTranslationTowards Universal Speech Discrete Tokens: A Case Study for ASR and TTS
Self-supervised learning (SSL) proficiency in speech-related tasks has driven research into utilizing discrete tokens for speech tasks like recognition and translation, which offer lower storage requirements and great po…
Self-Supervised Learningspeech-recognitionSpeech RecognitionSpeech SynthesisA Survey on Speech Large Language Models
Large Language Models (LLMs) exhibit strong contextual understanding and remarkable multitask performance. As a result, researchers have been actively exploring the integration of LLMs into the domain of speech understan…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Emotion RecognitionSpeech Emotion Recognition+6