JOOCI: a Framework for Learning Comprehensive Speech Representations
Information in speech can be divided into two categories: what is being said (content) and how it is expressed (other). Current state-of-the-art (SOTA) techniques model speech at fixed segments, usually 10-25 ms, using a single embedding. Given the orthogonal nature of other and content information, attempting to optimize both within a single embedding results in suboptimal solutions. This approach divides the models capacity, limiting its ability to build complex hierarchical features effectively. In this work, we present an end-to-end speech representation learning framework designed to jointly optimize the other and content information (JOOCI) in speech. By using separate learnable parameters, JOOCI addresses this optimization challenge by modeling other and content information independently. Our results show that JOOCI consistently outperforms other SOTA models of similar size (100 million parameters) and pre-training data used (960 hours) by a significant margin when evaluated on a range of speech downstream tasks in the SUPERB benchmark, as shown in Table 1.
Code (0)
등록된 구현이 없습니다.
Tasks
Representation LearningSpeech Representation LearningSimilar Papers 제목 키워드 기반
DM-Codec: Distilling Multimodal Representations for Speech Tokenization
Recent advancements in speech-language models have yielded significant improvements in speech tokenization and synthesis. However, effectively mapping the complex, multidimensional attributes of speech into discrete toke…
Self-Supervised LearningSpeech TokenizationTransfer the linguistic representations from TTS to accent conversion with non-parallel data
Accent conversion aims to convert the accent of a source speech to a target accent, meanwhile preserving the speaker's identity. This paper introduces a novel non-autoregressive framework for accent conversion that learn…
text-to-speechText to SpeechVoice ConversionLearning Disentangled Speech Representations
Disentangled representation learning in speech processing has lagged behind other domains, largely due to the lack of datasets with annotated generative factors for robust evaluation. To address this, we propose SynSpeec…
BenchmarkingDisentanglementInformativenessRepresentation Learning+1Textless Speech-to-Music Retrieval Using Emotion Similarity
We introduce a framework that recommends music based on the emotions of speech. In content creation and daily life, speech contains information about human emotions, which can be enhanced by music. Our framework focuses …
RetrievalES3: Evolving Self-Supervised Learning of Robust Audio-Visual Speech Representations
We propose a novel strategy ES3 for self-supervised learning of robust audio-visual speech representations from unlabeled talking face videos. While many recent approaches for this task primarily rely on guiding the …
Audio-Visual Speech RecognitionLipreadingSelf-Supervised LearningSpeech Recognition