CCStereo: Audio-Visual Contextual and Contrastive Learning for Binaural Audio Generation
Binaural audio generation (BAG) aims to convert monaural audio to stereo audio using visual prompts, requiring a deep understanding of spatial and semantic information. However, current models risk overfitting to room environments and lose fine-grained spatial details. In this paper, we propose a new audio-visual binaural generation model incorporating an audio-visual conditional normalisation layer that dynamically aligns the mean and variance of the target difference audio features using visual context, along with a new contrastive learning method to enhance spatial sensitivity by mining negative samples from shuffled visual features. We also introduce a cost-efficient way to utilise test-time augmentation in video data to enhance performance. Our approach achieves state-of-the-art generation accuracy on the FAIR-Play and MUSIC-Stereo benchmarks.
Code (0)
등록된 구현이 없습니다.
Tasks
Audio GenerationContrastive LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Visually Informed Binaural Audio Generation without Binaural Audios
Stereophonic audio, especially binaural audio, plays an essential role in immersive viewing environments. Recent research has explored generating visually guided stereophonic audios supervised by multi-channel audio coll…
Audio GenerationLocalize to Binauralize: Audio Spatialization From Visual Sound Source Localization
Videos with binaural audios provide an immersive viewing experience by enabling 3D sound sensation. Recent works attempt to generate binaural audio in a multimodal learning framework using large quantities of videos …
Audio GenerationSound Source LocalizationCyclic Learning for Binaural Audio Generation and Localization
Binaural audio is obtained by simulating the biological structure of human ears which plays an important role in artificial immersive spaces. A promising approach is to utilize mono audio and corresponding vision to …
Audio GenerationObjectObject LocalizationGeometry-Aware Multi-Task Learning for Binaural Audio Generation from Video
Binaural audio provides human listeners with an immersive spatial sound experience, but most existing videos lack binaural audio recordings. We propose an audio spatialization method that draws on visual information in v…
Audio GenerationMulti-Task LearningRoom Impulse Response (RIR)Points2Sound: From mono to binaural audio using 3D point cloud scenes
For immersive applications, the generation of binaural sound that matches its visual counterpart is crucial to bring meaningful experiences to people in a virtual environment. Recent studies have shown the possibility of…
Audio Synthesis