Analyzing Speech Unit Selection for Textless Speech-to-Speech Translation
Recent advancements in textless speech-to-speech translation systems have been driven by the adoption of self-supervised learning techniques. Although most state-of-the-art systems adopt a similar architecture to transform source language speech into sequences of discrete representations in the target language, the criteria for selecting these target speech units remains an open question. This work explores the selection process through a study of downstream tasks such as automatic speech recognition, speech synthesis, speaker recognition, and emotion recognition. Interestingly, our findings reveal a discrepancy in the optimization of discrete speech units: units that perform well in resynthesis performance do not necessarily correlate with those that enhance translation efficacy. This discrepancy underscores the nuanced complexity of target feature selection and its impact on the overall performance of speech-to-speech translation systems.
Code (0)
등록된 구현이 없습니다.
Tasks
Automatic Speech RecognitionEmotion Recognitionfeature selectionResynthesisSelf-Supervised LearningSpeaker Recognitionspeech-recognitionSpeech RecognitionSpeech SynthesisSpeech-to-Speech TranslationTranslationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
textless-lib: a Library for Textless Spoken Language Processing
Textless spoken language processing research aims to extend the applicability of standard NLP toolset onto spoken language and languages with few or no textual resources. In this paper, we introduce textless-lib, a PyTor…
ResynthesisImproving Textless Spoken Language Understanding with Discrete Units as Intermediate Target
Spoken Language Understanding (SLU) is a task that aims to extract semantic information from spoken utterances. Previous research has made progress in end-to-end SLU by using paired speech-text data, such as pre-trained …
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Few-Shot Learningspeech-recognition+2Textless Speech-to-Speech Translation on Real Data
We present a textless speech-to-speech translation (S2ST) system that can translate speech from one language into another language and can be built without the need of any text data. Different from existing work in the l…
Speech-to-Speech TranslationTranslationTextless Unit-to-Unit training for Many-to-Many Multilingual Speech-to-Speech Translation
This paper proposes a textless training method for many-to-many multilingual speech-to-speech translation that can also benefit the transfer of pre-trained knowledge to text-based systems, text-to-speech synthesis and te…
DecoderQuantizationRepresentation LearningSpeech Synthesis+7A Textless Metric for Speech-to-Speech Comparison
In this paper, we introduce a new and simple method for comparing speech utterances without relying on text transcripts. Our speech-to-speech comparison metric utilizes state-of-the-art speech2unit encoders like HuBERT t…
SentenceSpeech-to-Speech TranslationTranslation