Seeing Through Noise: Visually Driven Speaker Separation and Enhancement
Isolating the voice of a specific person while filtering out other voices or background noises is challenging when video is shot in noisy environments. We propose audio-visual methods to isolate the voice of a single speaker and eliminate unrelated sounds. First, face motions captured in the video are used to estimate the speaker's voice, by passing the silent video frames through a video-to-speech neural network-based model. Then the speech predictions are applied as a filter on the noisy input audio. This approach avoids using mixtures of sounds in the learning process, as the number of such possible mixtures is huge, and would inevitably bias the trained model. We evaluate our method on two audio-visual datasets, GRID and TCD-TIMIT, and show that our method attains significant SDR and PESQ improvements over the raw video-to-speech predictions, and a well-known audio-only method.
Code (0)
등록된 구현이 없습니다.
Tasks
Speaker SeparationSimilar Papers 제목 키워드 기반
Data-Driven Source Separation Based on Simplex Analysis
Blind source separation (BSS) is addressed, using a novel data-driven approach, based on a well-established probabilistic model. The proposed method is specifically designed for separation of multichannel audio mixtures.…
blind source separationSeeing through the Human Reporting Bias: Visual Classifiers from Noisy Human-Centric Labels
When human annotators are given a choice about what to label in an image, they apply their own subjective judgments on what to ignore and what to mention. We refer to these noisy "human-centric" annotations as exhibiting…
Image Captioningimage-classificationImage ClassificationMore than Words: In-the-Wild Visually-Driven Prosody for Text-to-Speech
In this paper we present VDTTS, a Visually-Driven Text-to-Speech model. Motivated by dubbing, VDTTS takes advantage of video frames as an additional input alongside text, and generates speech that matches the video signa…
text-to-speechText to SpeechVisualSpeaker: Visually-Guided 3D Avatar Lip Synthesis
Realistic, high-fidelity 3D facial animations are crucial for expressive avatar systems in human-computer interaction and accessibility. Although prior methods show promising quality, their reliance on the mesh domain li…
Automatic Speech RecognitionLip Readingspeech-recognitionSpeech Recognition+1VividVoice: A Unified Framework for Scene-Aware Visually-Driven Speech Synthesis
We introduce and define a novel task-Scene-Aware Visually-Driven Speech Synthesis, aimed at addressing the limitations of existing speech generation models in creating immersive auditory experiences that align with the r…
Speech Synthesis