Temporal-Spatial Neural Filter: Direction Informed End-to-End Multi-channel Target Speech Separation
Target speech separation refers to extracting the target speaker's speech from mixed signals. Despite the recent advances in deep learning based close-talk speech separation, the applications to real-world are still an open issue. Two main challenges are the complex acoustic environment and the real-time processing requirement. To address these challenges, we propose a temporal-spatial neural filter, which directly estimates the target speech waveform from multi-speaker mixture in reverberant environments, assisted with directional information of the speaker(s). Firstly, against variations brought by complex environment, the key idea is to increase the acoustic representation completeness through the jointly modeling of temporal, spectral and spatial discriminability between the target and interference source. Specifically, temporal, spectral, spatial along with the designed directional features are integrated to create a joint acoustic representation. Secondly, to reduce the latency, we design a fully-convolutional autoencoder framework, which is purely end-to-end and single-pass. All the feature computation is implemented by the network layers and operations to speed up the separation procedure. Evaluation is conducted on simulated reverberant dataset WSJ0-2mix and WSJ0-3mix under speaker-independent scenario. Experimental results demonstrate that the proposed method outperforms state-of-the-art deep learning based multi-channel approaches with fewer parameters and faster processing speed. Furthermore, the proposed temporal-spatial neural filter can handle mixtures with varying and unknown number of speakers and exhibits persistent performance even when existing a direction estimation error. Codes and models will be released soon.
Code (0)
등록된 구현이 없습니다.
Tasks
Speech SeparationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Numerical Gaussian process Kalman filtering for spatiotemporal systems
We present a novel Kalman filter for spatiotemporal systems called the numerical Gaussian process Kalman filter (GPKF). Numerical Gaussian processes have recently been introduced as a physics informed machine learning me…
BIG-bench Machine LearningGaussian ProcessesPhysics-informed machine learningRobust Noise Filtering in Image Sequences
Image sequences filtering have recently become a very important technical problem especially with the advent of new technology in multimedia and video systems applications. Often image sequences are corrupted by some amo…
Spatiotemporal Imputation with Graph-Informed Flow Matching
Missing data is a common challenge in spatiotemporal systems, arising in applications such as air quality monitoring and urban traffic management. Traditional machine learning approaches, like recurrent and graph neural …
Exploiting spatial information with the informed complex-valued spatial autoencoder for target speaker extraction
In conventional multichannel audio signal enhancement, spatial and spectral filtering are often performed sequentially. In contrast, it has been shown that for neural spatial filtering a joint approach of spectro-spatial…
PositionTarget Speaker ExtractionA Bioinspired Retinal Neural Network for Accurately Extracting Small-Target Motion Information in Cluttered Backgrounds
Robust and accurate detection of small moving targets in cluttered moving backgrounds is a significant and challenging problem for robotic visual systems to perform search and tracking tasks. Inspired by the neural circu…