Hierarchical Potential-based Reward Shaping from Task Specifications
The automatic synthesis of policies for robotic-control tasks through reinforcement learning relies on a reward signal that simultaneously captures many possibly conflicting requirements. In this paper, we in\-tro\-duce a novel, hierarchical, potential-based reward-shaping approach (HPRS) for defining effective, multivariate rewards for a large family of such control tasks. We formalize a task as a partially-ordered set of safety, target, and comfort requirements, and define an automated methodology to enforce a natural order among requirements and shape the associated reward. Building upon potential-based reward shaping, we show that HPRS preserves policy optimality. Our experimental evaluation demonstrates HPRS's superior ability in capturing the intended behavior, resulting in task-satisfying policies with improved comfort, and converging to optimal behavior faster than other state-of-the-art approaches. We demonstrate the practical usability of HPRS on several robotics applications and the smooth sim2real transition on two autonomous-driving scenarios for F1TENTH race cars.
Code (1)
Tasks
Autonomous DrivingReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Hierarchical Reward Design from Language: Enhancing Alignment of Agent Behavior with Human Specifications
When training artificial intelligence (AI) to perform tasks, humans often care not only about whether a task is completed but also how it is performed. As AI agents tackle increasingly complex tasks, aligning their behav…
Reinforcement LearningUnpacking Reward Shaping: Understanding the Benefits of Reward Engineering on Sample Complexity
Reinforcement learning provides an automated framework for learning behaviors from high-level reward specifications, but in practice the choice of reward function can be crucial for good results -- while in principle the…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Distributed Control using Reinforcement Learning with Temporal-Logic-Based Reward Shaping
We present a computational framework for synthesis of distributed control strategies for a heterogeneous team of robots in a partially observable environment. The goal is to cooperatively satisfy specifications given as …
reinforcement-learningReinforcement Learning (RL)Reward Shaping and Action Masking for Compositional Tasks using Behavior Trees and LLMs
Decomposing complex tasks into a sequence of simpler subtasks can improve learning efficiency for an autonomous agent. Reinforcement learning (RL) can be used to optimize agent policies to complete subtasks, but requires…
Reinforcement LearningSTO-RL: Offline RL under Sparse Rewards via LLM-Guided Subgoal Temporal Order
Offline reinforcement learning (RL) enables policy learning from pre-collected datasets, avoiding costly and risky online interactions, but it often struggles with long-horizon tasks involving sparse rewards. Existing go…
Reinforcement LearningOffline RL