Reward Shaping and Action Masking for Compositional Tasks using Behavior Trees and LLMs

2026-05-07 · Nicholas Potteiger, Ankita Samaddar, Taylor T. Johnson, Xenofon Koutsoukos arxiv

Decomposing complex tasks into a sequence of simpler subtasks can improve learning efficiency for an autonomous agent. Reinforcement learning (RL) can be used to optimize agent policies to complete subtasks, but requires well-defined subtask rewards and benefits from action masking. Recent work uses large language models (LLMs) to automate reward shaping and action masking, however none of them fully address reactivity to subtask failure and modularity to varying objects for compositional tasks. To overcome these challenges, we develop masking reward behavior tree (MRBT), a symbolic structure used as a reactive and modular reward and action mask function. We design an MRBT template and derive logical specifications to construct and verify MRBTs for a sequence of object-interaction subtasks. Further, we develop an automated pipeline that uses an LLM to generate MRBTs robust to varying task objects, an SMT-solver to verify correctness of specifications, and a neurosymbolic RL loop to train agents on compositional tasks. Experiments demonstrate successful generation and refinement of five MRBTs, consistently improving training efficiency and task success rates over baselines and MRBTs without action masking. We further highlight three advantages of MRBTs: transferability, modularity, and verifiability.

📄 PDF Abstract BibTeX arXiv:2605.05795

Code (0)

등록된 구현이 없습니다.

Tasks

Reinforcement Learning

The Mechanism Matters: When Knowledge Graphs Help Reinforcement Learning

2026-07-21 · Mohammed Sameer Syed arxiv

Knowledge graphs (KGs) are widely used to inject prior knowledge into reinforcement learning (RL), yet the literature is dominated by single-domain, positive-result method papers, so we lack a systematic account of when …

Reinforcement Learning Knowledge Graphs Offline RL

Reward Shaping via Meta-Learning

2019-01-27 · Haosheng Zou, Tongzheng Ren, Dong Yan, Hang Su 외

Reward shaping is one of the most effective methods to tackle the crucial yet challenging problem of credit assignment in Reinforcement Learning (RL). However, designing shaping functions usually requires much expert kno…

Meta-Learning Reinforcement Learning Reinforcement Learning (RL)

Shaping Advice in Deep Reinforcement Learning

2022-02-19 · Baicen Xiao, Bhaskar Ramasubramanian, Radha Poovendran

Reinforcement learning involves agents interacting with an environment to complete tasks. When rewards provided by the environment are sparse, agents may not receive immediate feedback on the quality of actions that they…

Deep Reinforcement Learning Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning+1

Shaping Advice in Deep Multi-Agent Reinforcement Learning

2021-03-29 · Baicen Xiao, Bhaskar Ramasubramanian, Radha Poovendran

Multi-agent reinforcement learning involves multiple agents interacting with each other and a shared environment to complete tasks. When rewards provided by the environment are sparse, agents may not receive immediate fe…

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Why Do Animals Need Shaping? A Theory of Task Composition and Curriculum Learning

2024-02-28 · Jin Hwa Lee, Stefano Sarao Mannelli, Andrew Saxe

Diverse studies in systems neuroscience begin with extended periods of curriculum training known as `shaping' procedures. These involve progressively studying component parts of more complex tasks, and can make the diffe…

Deep Reinforcement Learning reinforcement-learning Reinforcement Learning