Think While You Map: Asynchronous Vision-Language Agents for Incremental 3D Scene Graphs

2026-06-30 · Deniz Bickici, Michael Pabst, Shohei Mori, Dieter Schmalstieg arxiv

Open-vocabulary 3D scene graph methods typically operate in two stages: first reconstruct, then enrich with vision-language models, leaving the graph unqueryable during exploration. We argue that this sequential coupling is unnecessary and propose an asynchronous architecture in which lightweight online mapping runs concurrently with heavyweight semantic refinement. A probabilistic voxel-based backbone maintains stable object identities incrementally, while background VLM agents progressively enrich the graph. This framework resolves duplicate object tracks through semantic loop closure, attaches fine-grained visual attributes and derives spatial relations between objects. A multi-target frame scheduler amortizes VLM cost by selecting a small set of informative frames that jointly cover multiple targets. The resulting scene graph is queryable during exploration and grows in semantic richness over time. Our method matches or outperforms existing open-vocabulary 3D scene graph methods on semantic segmentation (ScanNet, Replica) and surpasses the prior state-of-the-art across three visual grounding benchmarks (Sr3D+, Nr3D, ScanRefer) by 15.3 to 18.8 A@0.25. Project page: https://denizbickici.github.io/thinkgraphs/

📄 PDF Abstract BibTeX arXiv:2606.31471

Code (0)

등록된 구현이 없습니다.

Tasks

Semantic Segmentation Visual Grounding

The Era of Agentic Organization: Learning to Organize with Language Models

2025-10-30 · Zewen Chi, Li Dong, Qingxiu Dong, Yaru Hao 외 arxiv

We envision a new era of AI, termed agentic organization, where agents solve complex problems by working collaboratively and concurrently, enabling outcomes beyond individual intelligence. To realize this vision, we intr…

Reinforcement Learning Mathematical Reasoning

Hume: Introducing System-2 Thinking in Visual-Language-Action Model

2025-05-27 · Haoming Song, Delin Qu, Yuanqi Yao, Qizhi Chen 외

Humans practice slow thinking before performing actual actions when handling complex tasks in the physical world. This thinking paradigm, recently, has achieved remarkable advancement in boosting Large Language Models (L…

Denoising Vision-Language-Action

TIC-VLA: A Think-in-Control Vision-Language-Action Model for Robot Navigation in Dynamic Environments

2026-02-02 · Zhiyu Huang, Yun Zhang, Johnson Liu, Rui Song 외 arxiv

Robots in dynamic, human-centric environments must follow language instructions while maintaining real-time reactive control. Vision-language-action (VLA) models offer a promising framework, but they assume temporally al…

Reinforcement Learning Robot Navigation

Asynchronous Reasoning: Training-Free Interactive Thinking LLMs

2025-12-11 · George Yakushev, Nataliia Babina, Masoud Vahid Dastgerdi, Vyacheslav Zhdanovskiy 외 arxiv

Many state-of-the-art LLMs are trained to think before giving their answer. Reasoning can greatly improve language model capabilities, but it also makes them less interactive: given a new input, a model must stop thinkin…

Think at 5 Hz, Act at 20 Hz: Asynchronous Fast-Slow Vision-Language-Action Inference for Closed-Loop Driving

2026-07-17 · Yun Li, Jiachen Gong, Simon Thompson, Ehsan Javanmardi 외 arxiv

Large language models bring instruction following and scene reasoning to end-to-end driving, but their inference latency collides with the control rate a vehicle requires. Existing closed-loop agents hide this gap by inv…

Instruction Following