Rejecting Hallucinated State Targets during Planning
Generative models can be used in planning to propose targets corresponding to states that agents deem either likely or advantageous to experience. However, imperfections, common in learned models, lead to infeasible hallucinated targets, which can cause delusional behaviors and thus safety concerns. This work first categorizes and investigates the properties of various kinds of infeasible targets. Then, we devise a strategy to reject infeasible targets with a generic target evaluator, which trains alongside planning agents as an add-on without the need to change the behavior nor the architectures of the agent (and the generative model) it is attached to. We highlight that, without proper training, the evaluator can produce delusional estimates, rendering the strategy futile. Thus, to learn correct evaluations of infeasible targets, we propose to use a combination of learning rule, architecture, and two assistive hindsight relabeling strategies. Our experiments validate significant reductions in delusional behaviors and enhancements in the performance of several kinds of existing planning agents.
Code (1)
Tasks
Decision MakingOut-of-Distribution GeneralizationSimilar Papers 제목 키워드 기반
Brain-Inspired Planning for Better Generalization in Reinforcement Learning
Existing Reinforcement Learning (RL) systems encounter significant challenges when applied to real-world scenarios, primarily due to poor generalization across environments that differ from their training conditions. Thi…
Reinforcement LearningAgent vs. Parametric World Models: Hybrid Planning for Reliable Language Agents
Language agents plan by generating not only actions but also implicit predictions of how the world will change. These imagined state updates make agents flexible, but they also create a distinct failure mode: hallucinate…
Hallucinating Value: A Pitfall of Dyna-style Planning with Imperfect Environment Models
Dyna-style reinforcement learning (RL) agents improve sample efficiency over model-free RL agents by updating the value function with simulated experience generated by an environment model. However, it is often difficult…
Reinforcement Learning (RL)Scalable Multi-Robot Informative Path Planning for Target Mapping via Deep Reinforcement Learning
Autonomous robots are widely utilized for mapping and exploration tasks due to their cost-effectiveness. Multi-robot systems offer scalability and efficiency, especially in terms of the number of robots deployed in more …
Collision AvoidanceDeep Reinforcement Learningreinforcement-learningReinforcement LearningHow Far Are LLMs from Symbolic Planners? An NLP-Based Perspective
The reasoning and planning abilities of Large Language Models (LLMs) have been a frequent topic of discussion in recent years. Their ability to take unstructured planning problems as input has made LLMs' integration into…