A Reminder of its Brittleness: Language Reward Shaping May Hinder Learning for Instruction Following Agents
Teaching agents to follow complex written instructions has been an important yet elusive goal. One technique for enhancing learning efficiency is language reward shaping (LRS). Within a reinforcement learning (RL) framework, LRS involves training a reward function that rewards behaviours precisely aligned with given language instructions. We argue that the apparent success of LRS is brittle, and prior positive findings can be attributed to weak RL baselines. Specifically, we identified suboptimal LRS designs that reward partially matched trajectories, and we characterised a novel reward perturbation to capture this issue using the concept of loosening task constraints. We provided theoretical and empirical evidence that agents trained using LRS rewards converge more slowly compared to pure RL agents. Our work highlights the brittleness of existing LRS methods, which has been overlooked in the previous studies.
Code (1)
Tasks
Instruction FollowingReinforcement Learning (RL)Similar Papers 제목 키워드 기반
ORSO: Accelerating Reward Design via Online Reward Selection and Policy Optimization
Reward shaping is critical in reinforcement learning (RL), particularly for complex tasks where sparse rewards can hinder learning. However, choosing effective shaping rewards from a set of reward functions in a computat…
continuous-controlContinuous ControlModel SelectionReinforcement Learning (RL)The Policy Cliff: A Theoretical Analysis of Reward-Policy Maps in Large Language Models
Reinforcement learning (RL) plays a crucial role in shaping the behavior of large language and reasoning models (LLMs/LRMs). However, it often produces brittle and unstable policies, leading to critical failures such as …
Reinforcement LearningComprehensive Overview of Reward Engineering and Shaping in Advancing Reinforcement Learning Applications
The aim of Reinforcement Learning (RL) in real-world applications is to create systems capable of making autonomous decisions by learning from their environment through trial and error. This paper emphasizes the importan…
Autonomous Drivingreinforcement-learningReinforcement LearningReinforcement Learning (RL)A Narration-based Reward Shaping Approach using Grounded Natural Language Commands
While deep reinforcement learning techniques have led to agents that are successfully able to learn to perform a number of tasks that had been previously unlearnable, these techniques are still susceptible to the longsta…
Deep Reinforcement LearningReinforcement LearningStarcraftStarcraft IIFrom Reward Shaping to Q-Shaping: Achieving Unbiased Learning with LLM-Guided Knowledge
Q-shaping is an extension of Q-value initialization and serves as an alternative to reward shaping for incorporating domain knowledge to accelerate agent training, thereby improving sample efficiency by directly shaping …
Language ModelingLanguage ModellingLarge Language Model