Neural Synthesis of Binaural Audio
We present a neural rendering approach for binaural sound synthesis that can produce realistic and spatially accurate binaural sound in realtime. The network takes, as input, a single-channel audio source and synthesizes, as output, two-channel binaural sound, conditioned on the relative position and orientation of the listener with respect to the source. We investigate deficiencies of the l2-loss on raw waveforms in a theoretical analysis and introduce an improved loss that overcomes these limitations. In an empirical evaluation, we establish that our approach is the first to generate spatially accurate waveform outputs (as measured by real recordings) and outperforms existing approaches by a considerable margin, both quantitatively and in a perceptual study. We will release a first-of-its-kind binaural audio dataset as a benchmark for future research.
Code (0)
등록된 구현이 없습니다.
Tasks
Neural RenderingPositionSimilar Papers 제목 키워드 기반
Zero-Shot Mono-to-Binaural Speech Synthesis
We present ZeroBAS, a neural method to synthesize binaural audio from monaural audio recordings and positional information without training on any binaural data. To our knowledge, this is the first published zero-shot ne…
Audio SynthesisDenoisingSpeech SynthesisZero-Shot LearningBinauralGrad: A Two-Stage Conditional Diffusion Probabilistic Model for Binaural Audio Synthesis
Binaural audio plays a significant role in constructing immersive augmented and virtual realities. As it is expensive to record binaural audio from the real world, synthesizing them from mono audio has attracted increasi…
Audio SynthesisPoints2Sound: From mono to binaural audio using 3D point cloud scenes
For immersive applications, the generation of binaural sound that matches its visual counterpart is crucial to bring meaningful experiences to people in a virtual environment. Recent studies have shown the possibility of…
Audio SynthesisArray2BR: An End-to-End Noise-immune Binaural Audio Synthesis from Microphone-array Signals
Telepresence technology aims to provide an immersive virtual presence for remote conference applications, and it is extremely important to synthesize high-quality binaural audio signals for this aim. Because the ambient …
Audio SynthesisDopplerBAS: Binaural Audio Synthesis Addressing Doppler Effect
Recently, binaural audio synthesis (BAS) has emerged as a promising research field for its applications in augmented and virtual realities. Binaural audio helps users orient themselves and establish immersion by providin…
Audio Synthesis