MASER: Modality-Adaptive Specialist Routing for Embodied 3D Spatial Intelligence

2026-06-01 · Hilton Raj, Vishnuram AV arxiv

In 3D environments, Embodied Agents answer spatially relevant questions through reasoning from a mixture of modalities including natural language, RGB images, point clouds, depth maps and camera poses. Existing Vision-Language models (VLMs) are fine-tuned over a single modality. This completely ignores the question semantics which may favor a different modality than the finetuned modality. To address this, we propose MASER (Modality-Adaptive SpEcialist Routing), a lightweight framework that trains five different modality adapters of a shared VLM backbone and learns a neural routing policy that selects the best adapter based on the question during inference. We encode each question with a frozen sentence transformer and pass the embedding through a small Multi-layer Perceptron (MLP) trained on oracle adapter-accuracy labels. We evaluate our methodology over the Open3D-VQA benchmark and our evaluations show that no single modality is universally optimal -- point-cloud answers are best in 51.5% of cases. MASER routes with 51.3% oracle agreement, outperforming a Random-Forest ablation (43.5%), with only a single adapter call per question.

📄 PDF Abstract BibTeX arXiv:2606.02463

Code (0)

등록된 구현이 없습니다.

Tasks

Point Clouds

MASER: Multi-Agent Reinforcement Learning with Subgoals Generated from Experience Replay Buffer

2022-06-20 · Jeewon Jeon, Woojun Kim, Whiyoung Jung, Youngchul Sung

In this paper, we consider cooperative multi-agent reinforcement learning (MARL) with sparse reward. To tackle this problem, we propose a novel method named MASER: MARL with subgoals generated from experience replay buff…

Multi-agent Reinforcement Learning Q-Learning reinforcement-learning Reinforcement Learning (RL)+2

SmartMage: Dynamic Modality Orchestration for 3D Scene Understanding

2026-08-05 · Yue Zhang, Yingzhao Jian, Yunqiu Xu, Xiaoxiao Sun 외 hf

Understanding 3D scenes is fundamental to embodied intelligence, requiring joint reasoning over heterogeneous information from multiple modalities, including visual and geometric cues. However, the relevance of these mod…

Multimodal Reasoning Scene Understanding

MedMix: Specialization-Consistent Federated Sparse MoEs under Modality Heterogeneity

2026-08-14 · Adiba Orzikulova, Dong Min Kim, Jaehong Yoon, Sung-Ju Lee arxiv

Federated multimodal medical AI faces modality heterogeneity at both the client and sample levels: clients may systematically lack access to specific modality types, while individual records within the same client may co…

Federated Learning

UniRoute: Unified Routing Mixture-of-Experts for Modality-Adaptive Remote Sensing Change Detection

2026-01-21 · Qingling Shu, Sibao Chen, Wei Lu, Zhihui You 외 arxiv

Current remote sensing change detection (CD) methods mainly rely on specialized models, which limits the scalability toward modality-adaptive Earth observation. For homogeneous CD, precise boundary delineation relies on …

Change Detection

KALAVAI: Predicting When Independent Specialist Fusion Works -- A Quantitative Model for Post-Hoc Cooperative LLM Training

2026-03-24 · Ramchand Kumaresan arxiv

Independently trained domain specialists can be fused post-hoc into a single model that outperforms any individual specialist, and the gain is predictable: gain = 0.82 x divergence - 2.72 (R^2 = 0.856, n=6, 3-26% diverge…