Learning Representations from Audio-Visual Spatial Alignment
We introduce a novel self-supervised pretext task for learning representations from audio-visual content. Prior work on audio-visual representation learning leverages correspondences at the video level. Approaches based on audio-visual correspondence (AVC) predict whether audio and video clips originate from the same or different video instances. Audio-visual temporal synchronization (AVTS) further discriminates negative pairs originated from the same video instance but at different moments in time. While these approaches learn high-quality representations for downstream tasks such as action recognition, their training objectives disregard spatial cues naturally occurring in audio and visual signals. To learn from these spatial cues, we tasked a network to perform contrastive audio-visual spatial alignment of 360{\deg} video and spatial audio. The ability to perform spatial alignment is enhanced by reasoning over the full spatial content of the 360{\deg} video using a transformer architecture to combine representations from multiple viewpoints. The advantages of the proposed pretext task are demonstrated on a variety of audio and visual downstream tasks, including audio-visual correspondence, spatial alignment, action recognition, and video semantic segmentation.
Code (0)
등록된 구현이 없습니다.
Tasks
Action RecognitionRepresentation LearningSemantic SegmentationVideo Semantic SegmentationSimilar Papers 제목 키워드 기반
Self-supervised Learning of Audio Representations from Audio-Visual Data using Spatial Alignment
Learning from audio-visual data offers many possibilities to express correspondence between the audio and visual content, similar to the human perception that relates aural and visual information. In this work, we presen…
Acoustic Scene ClassificationAction Recognitionobject-detectionObject Detection+4Towards Modeling the Interaction of Spatial-Associative Neural Network Representations for Multisensory Perception
Our daily perceptual experience is driven by different neural mechanisms that yield multisensory interaction as the interplay between exogenous stimuli and endogenous expectations. While the interaction of multisensory c…
Causal InferenceCAV-MAE Sync: Improving Contrastive Audio-Visual Mask Autoencoders via Fine-Grained Alignment
Recent advances in audio-visual learning have shown promising results in learning representations across modalities. However, most approaches rely on global audio representations that fail to capture fine-grained tempora…
audio-visual learningcross-modal alignmentRobust Audio-Visual Segmentation via Audio-Guided Visual Convergent Alignment
Accurately localizing audible objects based on audio-visual cues is the core objective of audio-visual segmentation. Most previous methods emphasize spatial or temporal multi-modal modeling, yet overlook challenges from …
Contrastive LearningSpatialV2A: Visual-Guided High-fidelity Spatial Audio Generation
While video-to-audio generation has achieved remarkable progress in semantic and temporal alignment, most existing studies focus solely on these aspects, paying limited attention to the spatial perception and immersive q…
Audio Generation