SceneAssistant: A Visual Feedback Agent for Open-Vocabulary 3D Scene Generation

2026-03-12 · Jun Luo, Jiaxiang Tang, Ruijie Lu, Gang Zeng arxiv

Text-to-3D scene generation from natural language is highly desirable for digital content creation. However, existing methods are largely domain-restricted or reliant on predefined spatial relationships, limiting their capacity for unconstrained, open-vocabulary 3D scene synthesis. In this paper, we introduce SceneAssistant, a visual-feedback-driven agent designed for open-vocabulary 3D scene generation. Our framework leverages modern 3D object generation model along with the spatial reasoning and planning capabilities of Vision-Language Models (VLMs). To enable open-vocabulary scene composition, we provide the VLMs with a comprehensive set of atomic operations (e.g., Scale, Rotate, FocusOn). At each interaction step, the VLM receives rendered visual feedback and takes actions accordingly, iteratively refining the scene to achieve more coherent spatial arrangements and better alignment with the input text. Experimental results demonstrate that our method can generate diverse, open-vocabulary, and high-quality 3D scenes. Both qualitative analysis and quantitative human evaluations demonstrate the superiority of our approach over existing methods. Furthermore, our method allows users to instruct the agent to edit existing scenes based on natural language commands. Our code is available at https://github.com/ROUJINN/SceneAssistant

📄 PDF Abstract BibTeX arXiv:2603.12238

Code (0)

등록된 구현이 없습니다.

Tasks

Spatial Reasoning Scene Generation

IndusAgent: Reinforcing Open-Vocabulary Industrial Anomaly Detection with Agentic Tools

2026-05-20 · Rongbin Tan, Fangfang Lin, Zhenlong Yuan, Min Qiu 외 arxiv

Multimodal large language models (MLLMs) have shown remarkable capability in bridging visual perception and textual reasoning, enabling zero-shot understanding across diverse industrial scenarios. However, their performa…

Anomaly Classification Reinforcement Learning Anomaly Detection

HM3D-OVON: A Dataset and Benchmark for Open-Vocabulary Object Goal Navigation

2024-09-22 · Naoki Yokoyama, Ram Ramrakhya, Abhishek Das, Dhruv Batra 외

We present the Habitat-Matterport 3D Open Vocabulary Object Goal Navigation dataset (HM3D-OVON), a large-scale benchmark that broadens the scope and semantic range of prior Object Goal Navigation (ObjectNav) benchmarks. …

Navigate Visual Navigation

RefineSVG: Visual Feedback-Driven Reinforcement Learning for Image-to-SVG Generation

2026-07-30 · Shaobo Liu, Feiqiao Mao, Shuaishuai Zhou, Yan Zhan 외 arxiv

We propose RefineSVG, a single-step closed-loop visual feedback framework that enables multimodal large language models (MLLMs) to perform high-fidelity image-to-SVG generation through self-correction. Existing MLLM-base…

Reinforcement Learning

Talking with Oompa Loompas: A novel framework for evaluating linguistic acquisition of LLM agents

2025-09-09 · Sankalp Tattwadarshi Swain, Anshika Krishnatray, Dhruv Kumar, Jagat Sesh Challa arxiv

Existing evaluation studies on linguistic competence of large language models (LLM agents) have focused primarily on vocabulary learning, morphological rule induction, syntactic generalization, pragmatic inference, and c…

Language Acquisition

Vision Harnessing Agent for Open Ad-hoc Segmentation

2026-05-19 · Zilin Wang, Stella X. Yu arxiv

Segmentation has become easy when the concept is known, requiring retrieval of a learned visual grounding from text. It remains hard for open ad-hoc concepts, where the grounding may not exist as one learned mask and mus…

Visual Grounding