Tracking and Planning with Spatial World Models
We introduce a method for real-time navigation and tracking with differentiably rendered world models. Learning models for control has led to impressive results in robotics and computer games, but this success has yet to be extended to vision-based navigation. To address this, we transfer advances in the emergent field of differentiable rendering to model-based control. We do this by planning in a learned 3D spatial world model, combined with a pose estimation algorithm previously used in the context of TSDF fusion, but now tailored to our setting and improved to incorporate agent dynamics. We evaluate over six simulated environments based on complex human-designed floor plans and provide quantitative results. We achieve up to 92% navigation success rate at a frequency of 15 Hz using only image and depth observations under stochastic, continuous dynamics.
Code (0)
등록된 구현이 없습니다.
Tasks
Pose EstimationSimilar Papers 제목 키워드 기반
An Informative Planning Framework for Target Tracking and Active Mapping in Dynamic Environments with ASVs
Mobile robot platforms are increasingly being used to automate information gathering tasks such as environmental monitoring. Efficient target tracking in dynamic environments is critical for applications such as search a…
Structured World Belief for Reinforcement Learning in POMDP
Object-centric world models provide structured representation of the scene and can be an important backbone in reinforcement learning and planning. However, existing approaches suffer in partially-observable environments…
Inductive BiasObjectObject Trackingreinforcement-learning+2MineAnyBuild: Benchmarking Spatial Planning for Open-world AI Agents
Spatial Planning is a crucial part in the field of spatial intelligence, which requires the understanding and planning about object arrangements in space perspective. AI agents with the spatial planning ability can bette…
BenchmarkingMinecraftQuestion AnsweringSpatial Reasoning+2ResWorld: Temporal Residual World Model for End-to-End Autonomous Driving
The comprehensive understanding capabilities of world models for driving scenarios have significantly improved the planning accuracy of end-to-end autonomous driving frameworks. However, the redundant modeling of static …
Autonomous DrivingTracking Meets Large Multimodal Models for Driving Scenario Understanding
Large Multimodal Models (LMMs) have recently gained prominence in autonomous driving research, showcasing promising capabilities across various emerging benchmarks. LMMs specifically designed for this domain have demonst…
Autonomous Driving