wav2pos: Sound Source Localization using Masked Autoencoders
We present a novel approach to the 3D sound source localization task for distributed ad-hoc microphone arrays by formulating it as a set-to-set regression problem. By training a multi-modal masked autoencoder model that operates on audio recordings and microphone coordinates, we show that such a formulation allows for accurate localization of the sound source, by reconstructing coordinates masked in the input. Our approach is flexible in the sense that a single model can be used with an arbitrary number of microphones, even when a subset of audio recordings and microphone coordinates are missing. We test our method on simulated and real-world recordings of music and speech in indoor environments, and demonstrate competitive performance compared to both classical and other learning based localization methods.
Code (1)
Tasks
Indoor LocalizationSound Source LocalizationSimilar Papers 제목 키워드 기반
Masked Autoencoders for Egocentric Video Understanding @ Ego4D Challenge 2022
In this report, we present our approach and empirical results of applying masked autoencoders in two egocentric video understanding tasks, namely, Object State Change Classification and PNR Temporal Localization, of Ego4…
Object State Change ClassificationTemporal LocalizationVideo UnderstandingAuralNet: Hierarchical Attention-based 3D Binaural Localization of Overlapping Speakers
We propose AuralNet, a novel 3D multi-source binaural sound source localization approach that localizes overlapping sources in both azimuth and elevation without prior knowledge of the number of sources. AuralNet employs…
Sound Source LocalizationFocus on Texture: Rethinking Pre-training in Masked Autoencoders for Medical Image Classification
Masked Autoencoders (MAEs) have emerged as a dominant strategy for self-supervised representation learning in natural images, where models are pre-trained to reconstruct masked patches with a pixel-wise mean squared erro…
Gallbladder Cancer DetectionMedical Image ClassificationUnsupervised Pre-trainingRepresentation LearningMasked Autoencoders for Ultrasound Signals: Robust Representation Learning for Downstream Applications
We investigated the adaptation and performance of Masked Autoencoders (MAEs) with Vision Transformer (ViT) architectures for self-supervised representation learning on one-dimensional (1D) ultrasound signals. Although MA…
Self-Supervised LearningRepresentation LearningSound Source Localization is All about Cross-Modal Alignment
Humans can easily perceive the direction of sound sources in a visual scene, termed sound source localization. Recent studies on learning-based sound source localization have mainly explored the problem from a localizati…
Allcross-modal alignmentCross-Modal RetrievalRetrieval+1