Real-time Audio Video Enhancement \\with a Microphone Array and Headphones
This paper presents a complete hardware and software pipeline for real-time speech enhancement in noisy and reverberant conditions. The device consists of a microphone array and a camera mounted on eyeglasses, connected to an embedded system that enhances speech and plays back the audio in headphones, with a latency of maximum 120 msec. The proposed approach relies on face detection, tracking and verification to enhance the speech of a target speaker using a beamformer and a postfiltering neural network. Results demonstrate the feasibility of the approach, and opens the door to the exploration and validation of a wide range of beamformer and speech enhancement methods for real-time speech enhancement.
Code (0)
등록된 구현이 없습니다.
Tasks
Face DetectionSpeech EnhancementVideo EnhancementSimilar Papers 제목 키워드 기반
Cooperative Audio Source Separation and Enhancement Using Distributed Microphone Arrays and Wearable Devices
Augmented listening devices such as hearing aids often perform poorly in noisy and reverberant environments with many competing sound sources. Large distributed microphone arrays can improve performance, but data from re…
Audio Source SeparationReal-Time System for Audio-Visual Target Speech Enhancement
We present a live demonstration for RAVEN, a real-time audio-visual speech enhancement system designed to run entirely on a CPU. In single-channel, audio-only settings, speech enhancement is traditionally approached as t…
Audio-Visual Speech RecognitionSpeech EnhancementUtterance-Wise Meeting Transcription System Using Asynchronous Distributed Microphones
A novel framework for meeting transcription using asynchronous microphones is proposed in this paper. It consists of audio synchronization, speaker diarization, utterance-wise speech enhancement using guided source separ…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)speaker-diarizationSpeaker Diarization+3A Causal U-net based Neural Beamforming Network for Real-Time Multi-Channel Speech Enhancement
People are meeting through video conferencing more often. While single channel speech enhancement techniques are useful for the individual participants, the speech quality will be significantly degraded in large meeting …
CPUSpeech EnhancementMulti-Microphone and Multi-Modal Emotion Recognition in Reverberant Environment
This paper presents a Multi-modal Emotion Recognition (MER) system designed to enhance emotion recognition accuracy in challenging acoustic conditions. Our approach combines a modified and extended Hierarchical Token-sem…
Emotion Recognition