Space-Time Memory Network for Sounding Object Localization in Videos
Leveraging temporal synchronization and association within sight and sound is an essential step towards robust localization of sounding objects. To this end, we propose a space-time memory network for sounding object localization in videos. It can simultaneously learn spatio-temporal attention over both uni-modal and cross-modal representations from audio and visual modalities. We show and analyze both quantitatively and qualitatively the effectiveness of incorporating spatio-temporal learning in localizing audio-visual objects. We demonstrate that our approach generalizes over various complex audio-visual scenes and outperforms recent state-of-the-art methods.
Code (0)
등록된 구현이 없습니다.
Tasks
Object LocalizationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Cyclic Learning for Binaural Audio Generation and Localization
Binaural audio is obtained by simulating the biological structure of human ears which plays an important role in artificial immersive spaces. A promising approach is to utilize mono audio and corresponding vision to …
Audio GenerationObjectObject LocalizationDiscriminative Sounding Objects Localization via Self-supervised Audiovisual Matching
Discriminatively localizing sounding objects in cocktail-party, i.e., mixed sound scenes, is commonplace for humans, but still challenging for machines. In this paper, we propose a two-stage learning framework to perform…
ObjectObject LocalizationClass-aware Sounding Objects Localization via Audiovisual Correspondence
Audiovisual scenes are pervasive in our daily life. It is commonplace for humans to discriminatively localize different sounding objects but quite challenging for machines to achieve class-aware sounding objects localiza…
Objectobject-detectionObject DetectionObject Localization+1Localizing Visual Sounds the Easy Way
Unsupervised audio-visual source localization aims at localizing visible sound sources in a video without relying on ground-truth localization for training. Previous works often seek high audio-visual similarities for li…
Multi-scale Multi-instance Visual Sound Localization and Segmentation
Visual sound localization is a typical and challenging problem that predicts the location of objects corresponding to the sound source in a video. Previous methods mainly used the audio-visual association between global …
Object Localization