Cliff Diving: Exploring Reward Surfaces in Reinforcement Learning Environments

2022-05-14 · Ryan Sullivan, J. K. Terry, Benjamin Black, John P. Dickerson

Visualizing optimization landscapes has led to many fundamental insights in numeric optimization, and novel improvements to optimization techniques. However, visualizations of the objective that reinforcement learning optimizes (the "reward surface") have only ever been generated for a small number of narrow contexts. This work presents reward surfaces and related visualizations of 27 of the most widely used reinforcement learning environments in Gym for the first time. We also explore reward surfaces in the policy gradient direction and show for the first time that many popular reinforcement learning environments have frequent "cliffs" (sudden large drops in expected return). We demonstrate that A2C often "dives off" these cliffs into low reward regions of the parameter space while PPO avoids them, confirming a popular intuition for PPO's improved performance over previous methods. We additionally introduce a highly extensible library that allows researchers to easily generate these visualizations in the future. Our findings provide new intuition to explain the successes and failures of modern RL methods, and our visualizations concretely characterize several failure modes of reinforcement learning agents in novel ways.

📄 PDF Abstract BibTeX arXiv:2205.07015

Code (0)

등록된 구현이 없습니다.

Tasks

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

A2C A2C, or Advantage Actor Critic, is a synchronous version of the A3C policy gradient method. As an alternative to the asynchronous…

Entropy Regularization 설명 없음

PPO Proximal Policy Optimization, or PPO, is a policy gradient method for reinforcement learning. The motivation was to have an algorithm with the data efficiency and reliable…

Cliff: Learning Process Rewards from the First Mistake

2026-09-02 · Peixuan Han, Runhui Wang, Ketan Ramaneti, Jie Hao 외 hf

Reinforcement learning with verifiable rewards (RLVR) has emerged as a powerful paradigm for large language model (LLM) post-training, but its reliance on coarse outcome rewards leads to limited guidance on intermediate …

Reinforcement Learning

LoRA Scaffolded Policy Optimization (LSPO): A Sampling-Time Low-Rank Scaffold for Recovering Reinforcement-Learning Gradient on Zero-Reward Cliff Prompts

2026-07-30 · Ken Ding arxiv

Reinforcement learning from verifiable rewards (RLVR) for mathematical reasoning suffers from a structural blind spot: on "cliff" prompts-those on which every sampled rollout in a group fails-the group-normalized advanta…

Reinforcement Learning Mathematical Reasoning

Visceral Machines: Risk-Aversion in Reinforcement Learning with Intrinsic Physiological Rewards

2018-05-25 · Daniel McDuff, Ashish Kapoor

As people learn to navigate the world, autonomic nervous system (e.g., "fight or flight") responses provide intrinsic feedback about the potential consequence of action choices (e.g., becoming nervous when close to a cli…

Navigate reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Visceral Machines: Reinforcement Learning with Intrinsic Physiological Rewards

2019-05-01 · ICLR 2019 5 · Daniel McDuff, Ashish Kapoor

The human autonomic nervous system has evolved over millions of years and is essential for survival and responding to threats. As people learn to navigate the world, ``fight or flight'' responses provide intrinsic feedb…