Reward Tweaking: Maximizing the Total Reward While Planning for Short Horizons

2020-02-09 · Chen Tessler, Shie Mannor

In reinforcement learning, the discount factor $\gamma$ controls the agent's effective planning horizon. Traditionally, this parameter was considered part of the MDP; however, as deep reinforcement learning algorithms tend to become unstable when the effective planning horizon is long, recent works refer to $\gamma$ as a hyper-parameter -- thus changing the underlying MDP and potentially leading the agent towards sub-optimal behavior on the original task. In this work, we introduce \emph{reward tweaking}. Reward tweaking learns a surrogate reward function $\tilde r$ for the discounted setting that induces optimal behavior on the original finite-horizon total reward task. Theoretically, we show that there exists a surrogate reward that leads to optimality in the original task and discuss the robustness of our approach. Additionally, we perform experiments in high-dimensional continuous control tasks and show that reward tweaking guides the agent towards better long-horizon returns although it plans for short horizons.

📄 PDF Abstract BibTeX arXiv:2002.03327

Code (0)

등록된 구현이 없습니다.

Tasks

continuous-control Continuous ControlDeep Reinforcement Learningreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Analyzing and Bridging the Gap between Maximizing Total Reward and Discounted Reward in Deep Reinforcement Learning

2024-07-18 · Shuyu Yin, Fei Wen, Peilin Liu, Tao Luo

In deep reinforcement learning applications, maximizing discounted reward is often employed instead of maximizing total reward to ensure the convergence and stability of algorithms, even though the performance metric for…

Deep Reinforcement Learning reinforcement-learning Reinforcement Learning

Maximal Objectives in the Multi-armed Bandit with Applications

2020-06-11 · Eren Ozbay, Vijay Kamble

In several applications of the stochastic multi-armed bandit problem, the traditional objective of maximizing the expected total reward can be inappropriate. In this paper, motivated by certain operational concerns in on…

Decision Making

Hindsight Reward Tweaking via Conditional Deep Reinforcement Learning

2021-09-06 · Ning Wei, Jiahua Liang, Di Xie, ShiLiang Pu

Designing optimal reward functions has been desired but extremely difficult in reinforcement learning (RL). When it comes to modern complex tasks, sophisticated reward functions are widely used to simplify policy learnin…

Deep Reinforcement Learning MuJoCo reinforcement-learning Reinforcement Learning+1

Reinforcement Learning with Ex-Post Max-Min Fairness

2021-09-29 · Wang Chi Cheung, Zi Yi Ewe

We consider reinforcement learning with vectorial rewards, where the agent receives a vector of $K\geq 2$ different types of rewards at each time step. The agent aims to maximize the minimum total reward among the $K$ re…

Fairness reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Non-maximizing policies that fulfill multi-criterion aspirations in expectation

2024-08-08 · Simon Dima, Simon Fischer, Jobst Heitzig, Joss Oliver

In dynamic programming and reinforcement learning, the policy for the sequential decision making of an agent in a stochastic environment is usually determined by expressing the goal as a scalar reward function and seekin…

Sequential Decision Making