PhaseCoder: Microphone Geometry-Agnostic Spatial Audio Understanding for Multimodal LLMs
Current multimodal LLMs process audio as a mono stream, ignoring the rich spatial information essential for embodied AI. Existing spatial audio models, conversely, are constrained to fixed microphone geometries, preventing deployment across diverse devices. We present PhaseCoder, a transformer-only spatial audio encoder that is agnostic to microphone geometry. PhaseCoder takes raw multichannel audio and microphone coordinates as inputs to perform localization and produces robust spatial embeddings. We demonstrate that Gemma 3n LLM can be fine-tuned to reason over "Spatial Audio Tokens" produced by PhaseCoder. We show our encoder achieves state-of-the-art results on microphone-invariant localization benchmarks and, for the first time, enables an LLM to perform complex spatial reasoning and targeted transcription tasks from an arbitrary microphone array.
Code (0)
등록된 구현이 없습니다.
Tasks
Spatial ReasoningSimilar Papers 제목 키워드 기반
One model to enhance them all: array geometry agnostic multi-channel personalized speech enhancement
With the recent surge of video conferencing tools usage, providing high-quality speech signals and accurate captions have become essential to conduct day-to-day business or connect with friends and families. Single-chann…
AllAutomatic Speech RecognitionAutomatic Speech Recognition (ASR)Speech Enhancement+2Multi-Geometry Spatial Acoustic Modeling for Distant Speech Recognition
The use of spatial information with multiple microphones can improve far-field automatic speech recognition (ASR) accuracy. However, conventional microphone array techniques degrade speech enhancement performance when th…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Deep ClusteringDistant Speech Recognition+3HARP: A Large-Scale Higher-Order Ambisonic Room Impulse Response Dataset
This contribution introduces a dataset of 7th-order Ambisonic Room Impulse Responses (HOA-RIRs), created using the Image Source Method. By employing higher-order Ambisonics, our dataset enables precise spatial audio repr…
ArrayDPS: Unsupervised Blind Speech Separation with a Diffusion Prior
Blind Speech Separation (BSS) aims to separate multiple speech sources from audio mixtures recorded by a microphone array. The problem is challenging because it is a blind inverse problem, i.e., the microphone array geom…
Room Impulse Response (RIR)Speech SeparationTowards Accurate and Robust Surveillance Roadside IVD via Trackletized Audio-Visual Reasoning
Idling Vehicle Detection (IVD) seeks to determine, at the final frame of a video clip, whether any vehicle is idling, meaning the vehicle is stationary with its engine running, using synchronized video from a remote surv…
Multi-Object TrackingVisual Reasoning