BusyBot: Learning to Interact, Reason, and Plan in a BusyBoard Environment
We introduce BusyBoard, a toy-inspired robot learning environment that leverages a diverse set of articulated objects and inter-object functional relations to provide rich visual feedback for robot interactions. Based on this environment, we introduce a learning framework, BusyBot, which allows an agent to jointly acquire three fundamental capabilities (interaction, reasoning, and planning) in an integrated and self-supervised manner. With the rich sensory feedback provided by BusyBoard, BusyBot first learns a policy to efficiently interact with the environment; then with data collected using the policy, BusyBot reasons the inter-object functional relations through a causal discovery network; and finally by combining the learned interaction policy and relation reasoning skill, the agent is able to perform goal-conditioned manipulation tasks. We evaluate BusyBot in both simulated and real-world environments, and validate its generalizability to unseen objects and relations. Video is available at https://youtu.be/EJ98xBJZ9ek.
Code (1)
Tasks
Causal DiscoveryRobot ManipulationRobot Task PlanningScene Graph GenerationSimilar Papers 제목 키워드 기반
ReasonPlanner: Enhancing Autonomous Planning in Dynamic Environments with Temporal Knowledge Graphs and LLMs
Planning and performing interactive tasks, such as conducting experiments to determine the melting point of an unknown substance, is straightforward for humans but poses significant challenges for autonomous agents. We i…
Knowledge GraphsChat with the Environment: Interactive Multimodal Perception Using Large Language Models
Programming robot behavior in a complex world faces challenges on multiple levels, from dextrous low-level skills to high-level planning and reasoning. Recent pre-trained Large Language Models (LLMs) have shown remarkabl…
Thinking by Doing: Building Efficient World Model Reasoning in LLMs via Multi-turn Interaction
Developing robust world model reasoning is crucial for large language model (LLM) agents to plan and interact in complex environments. While multi-turn interaction offers a superior understanding of environmental dynamic…
Active LearningPlanU: Large Language Model Reasoning through Planning under Uncertainty
Large Language Models (LLMs) are increasingly being explored across a range of reasoning tasks. However, LLMs sometimes struggle with reasoning tasks under uncertainty that are relatively easy for humans, such as plannin…
Navigating Motion Agents in Dynamic and Cluttered Environments through LLM Reasoning
This paper advances motion agents empowered by large language models (LLMs) toward autonomous navigation in dynamic and cluttered environments, significantly surpassing first and recent seminal but limited studies on LLM…
Autonomous NavigationMotion GenerationSpatial Reasoning