Disentangling speech from surroundings with neural embeddings
We present a method to separate speech signals from noisy environments in the embedding space of a neural audio codec. We introduce a new training procedure that allows our model to produce structured encodings of audio waveforms given by embedding vectors, where one part of the embedding vector represents the speech signal, and the rest represent the environment. We achieve this by partitioning the embeddings of different input waveforms and training the model to faithfully reconstruct audio from mixed partitions, thereby ensuring each partition encodes a separate audio attribute. As use cases, we demonstrate the separation of speech from background noise or from reverberation characteristics. Our method also allows for targeted adjustments of the audio output characteristics.
Code (0)
등록된 구현이 없습니다.
Tasks
AttributeSimilar Papers 제목 키워드 기반
Fine-Grained Action Retrieval Through Multiple Parts-of-Speech Embeddings
We address the problem of cross-modal fine-grained action retrieval between text and video. Cross-modal retrieval is commonly achieved through learning a shared embedding space, that can indifferently embed modalities. I…
Cross-Modal RetrievalPOSRetrievalTAG+1Disentangling Speakers in Multi-Talker Speech Recognition with Speaker-Aware CTC
Multi-talker speech recognition (MTASR) faces unique challenges in disentangling and transcribing overlapping speech. To address these challenges, this paper investigates the role of Connectionist Temporal Classification…
Disentanglementspeech-recognitionSpeech RecognitionDisentangling dialects: a neural approach to Indo-Aryan historical phonology and subgrouping
This paper seeks to uncover patterns of sound change across Indo-Aryan languages using an LSTM encoder-decoder architecture. We augment our models with embeddings represent-ing language ID, part of speech, and other feat…
DecoderWord EmbeddingsAdvancing the dimensionality reduction of speaker embeddings for speaker diarisation: disentangling noise and informing speech activity
The objective of this work is to train noise-robust speaker embeddings adapted for speaker diarisation. Speaker embeddings play a crucial role in the performance of diarisation systems, but they often capture spurious in…
Dimensionality ReductionExplainable Disentanglement on Discrete Speech Representations for Noise-Robust ASR
Discrete audio representations are gaining traction in speech modeling due to their interpretability and compatibility with large language models, but are not always optimized for noisy or real-world environments. Buildi…