Exploration-Exploitation Trade-off in Reinforcement Learning on Online Markov Decision Processes with Global Concave Rewards

2019-05-15 · Wang Chi Cheung

We consider an agent who is involved in a Markov decision process and receives a vector of outcomes every round. Her objective is to maximize a global concave reward function on the average vectorial outcome. The problem models applications such as multi-objective optimization, maximum entropy exploration, and constrained optimization in Markovian environments. In our general setting where a stationary policy could have multiple recurrent classes, the agent faces a subtle yet consequential trade-off in alternating among different actions for balancing the vectorial outcomes. In particular, stationary policies are in general sub-optimal. We propose a no-regret algorithm based on online convex optimization (OCO) tools (Agrawal and Devanur 2014) and UCRL2 (Jaksch et al. 2010). Importantly, we introduce a novel gradient threshold procedure, which carefully controls the switches among actions to handle the subtle trade-off. By delaying the gradient updates, our procedure produces a non-stationary policy that diversifies the outcomes for optimizing the objective. The procedure is compatible with a variety of OCO tools.

📄 PDF Abstract BibTeX arXiv:1905.06466

Code (0)

등록된 구현이 없습니다.

Tasks

Reinforcement Learning

Reward Shaping via Diffusion Process in Reinforcement Learning

2023-06-20 · Peeyush Kumar

Reinforcement Learning (RL) models have continually evolved to navigate the exploration - exploitation trade-off in uncertain Markov Decision Processes (MDPs). In this study, I leverage the principles of stochastic therm…

Navigate reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

The Role of Exploration for Task Transfer in Reinforcement Learning

2022-10-11 · Jonathan C Balloch, Julia Kim, and Jessica L Inman, Mark O Riedl

The exploration--exploitation trade-off in reinforcement learning (RL) is a well-known and much-studied problem that balances greedy action selection with novel experience, and the study of exploration methods is usually…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Transfer Learning

Reward Shifting for Optimistic Exploration and Conservative Exploitation

2021-09-29 · Hao Sun, Lei Han, Jian Guo, Bolei Zhou

In this work, we study the simple yet universally applicable case of reward shaping, the linear transformation, in value-based Deep Reinforcement Learning. We show that reward shifting, as the simplest linear reward tran…

continuous-control Continuous Control Deep Reinforcement Learning Offline RL

Exploration Conscious Reinforcement Learning Revisited

2018-12-13 · Lior Shani, Yonathan Efroni, Shie Mannor

The Exploration-Exploitation tradeoff arises in Reinforcement Learning when one cannot tell if a policy is optimal. Then, there is a constant need to explore new actions instead of exploiting past experience. In practice…

Deep Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

DiPO: Disentangled Perplexity Policy Optimization for Fine-grained Exploration-Exploitation Trade-Off

2026-04-15 · Xiaofan Li, Ming Yang, Zhiyuan Ma, Shichao Ma 외 arxiv

Reinforcement Learning with Verifiable Rewards (RLVR) has catalyzed significant advances in the reasoning capabilities of Large Language Models (LLMs). However, effectively managing the exploration and exploitation trade…

Reinforcement Learning Mathematical Reasoning