Putting a Face to the Voice: Fusing Audio and Visual Signals Across a Video to Determine Speakers
In this paper, we present a system that associates faces with voices in a video by fusing information from the audio and visual signals. The thesis underlying our work is that an extremely simple approach to generating (weak) speech clusters can be combined with visual signals to effectively associate faces and voices by aggregating statistics across a video. This approach does not need any training data specific to this task and leverages the natural coherence of information in the audio and visual streams. It is particularly applicable to tracking speakers in videos on the web where a priori information about the environment (e.g., number of speakers, spatial signals for beamforming) is not available. We performed experiments on a real-world dataset using this analysis framework to determine the speaker in a video. Given a ground truth labeling determined by human rater consensus, our approach had ~71% accuracy.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
VoiceVector: Multimodal Enrolment Vectors for Speaker Separation
We present a transformer-based architecture for voice separation of a target speaker from multiple other speakers and ambient noise. We achieve this by using two separate neural networks: (A) An enrolment network designe…
Speaker SeparationMore than Text: Multi-modal Chinese Word Segmentation
Chinese word segmentation (CWS) is undoubtedly an important basic task in natural language processing. Previous works only focus on the textual modality, but there are often audio and video utterances (such as news broad…
Chinese Word SegmentationFace2VoiceSync: Lightweight Face-Voice Consistency for Text-Driven Talking Face Generation
Recent studies in speech-driven talking face generation achieve promising results, but their reliance on fixed-driven speech limits further applications (e.g., face-voice mismatch). Thus, we extend the task to a more cha…
Talking Face GenerationFace AlignmentVocaLiST: An Audio-Visual Synchronisation Model for Lips and Voices
In this paper, we address the problem of lip-voice synchronisation in videos containing human face and voice. Our approach is based on determining if the lips motion and the voice in a video are synchronised or not, depe…
Audio-Visual SynchronizationMusic Source SeparationVoViT: Low Latency Graph-based Audio-Visual Voice Separation Transformer
This paper presents an audio-visual approach for voice separation which produces state-of-the-art results at a low latency in two scenarios: speech and singing voice. The model is based on a two-stage network. Motion cue…
Speech Separation