Bridging Scene Understanding and Task Execution with Flexible Simulation Environments

2020-11-20 · Zachary Ravichandran, J. Daniel Griffith, Benjamin Smith, Costas Frost

Significant progress has been made in scene understanding which seeks to build 3D, metric and object-oriented representations of the world. Concurrently, reinforcement learning has made impressive strides largely enabled by advances in simulation. Comparatively, there has been less focus in simulation for perception algorithms. Simulation is becoming increasingly vital as sophisticated perception approaches such as metric-semantic mapping or 3D dynamic scene graph generation require precise 3D, 2D, and inertial information in an interactive environment. To that end, we present TESSE (Task Execution with Semantic Segmentation Environments), an open source simulator for developing scene understanding and task execution algorithms. TESSE has been used to develop state-of-the-art solutions for metric-semantic mapping and 3D dynamic scene graph generation. Additionally, TESSE served as the platform for the GOSEEK Challenge at the International Conference of Robotics and Automation (ICRA) 2020, an object search competition with an emphasis on reinforcement learning. Code for TESSE is available at https://github.com/MIT-TESSE.

📄 PDF Abstract BibTeX arXiv:2011.10452

Code (0)

등록된 구현이 없습니다.

Tasks

Graph Generation reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Scene Graph Generation Scene Understanding Semantic Segmentation

Vision-Language-Policy Model for Dynamic Robot Task Planning

2025-12-22 · Jin Wang, Kim Tien Ly, Jacques Cloete, Nikos Tsagarakis 외 arxiv

Bridging the gap between natural language commands and autonomous execution in unstructured environments remains an open challenge for robotics. This requires robots to perceive and reason over the current task scene thr…

Robot Task Planning

VLM-FO1: Bridging the Gap Between High-Level Reasoning and Fine-Grained Perception in VLMs

2025-09-30 · Peng Liu, Haozhan Shen, Chunxin Fang, Zhicheng Sun 외 arxiv

Vision-Language Models (VLMs) excel at high-level scene understanding but falter on fine-grained perception tasks requiring precise localization. This failure stems from a fundamental mismatch, as generating exact numeri…

Scene Understanding Visual Grounding

Bridging Discrete Planning and Continuous Execution for Redundant Robot

2026-04-02 · Teng Yan, Yue Yu, Yihan Liu, Bingzhuo Zhong arxiv

Voxel-grid reinforcement learning is widely adopted for path planning in redundant manipulators due to its simplicity and reproducibility. However, direct execution through point-wise numerical inverse kinematics on 7-Do…

Reinforcement Learning

Training-Free Hierarchical Scene Understanding for Gaussian Splatting with Superpoint Graphs

2025-04-17 · Shaohui Dai, Yansong Qu, Zheyan Li, Xinyang Li 외

Bridging natural language and 3D geometry is a crucial step toward flexible, language-driven scene understanding. While recent advances in 3D Gaussian Splatting (3DGS) have enabled fast and high-quality scene reconstruct…

3D geometry 3DGS Scene Understanding

Go-SLAM: Grounded Object Segmentation and Localization with Gaussian Splatting SLAM

2024-09-25 · Phu Pham, Dipam Patel, Damon Conover, Aniket Bera

We introduce Go-SLAM, a novel framework that utilizes 3D Gaussian Splatting SLAM to reconstruct dynamic environments while embedding object-level information within the scene representations. This framework employs advan…

3D Scene Reconstruction Object Semantic Segmentation