Model-Free Robust Average-Reward Reinforcement Learning

2023-05-17 · Yue Wang, Alvaro Velasquez, George Atia, Ashley Prater-Bennette, Shaofeng Zou

Robust Markov decision processes (MDPs) address the challenge of model uncertainty by optimizing the worst-case performance over an uncertainty set of MDPs. In this paper, we focus on the robust average-reward MDPs under the model-free setting. We first theoretically characterize the structure of solutions to the robust average-reward Bellman equation, which is essential for our later convergence analysis. We then design two model-free algorithms, robust relative value iteration (RVI) TD and robust RVI Q-learning, and theoretically prove their convergence to the optimal solution. We provide several widely used uncertainty sets as examples, including those defined by the contamination model, total variation, Chi-squared divergence, Kullback-Leibler (KL) divergence and Wasserstein distance.

📄 PDF Abstract BibTeX arXiv:2305.10504

Code (0)

등록된 구현이 없습니다.

Tasks

modelQ-Learningreinforcement-learning Reinforcement Learning

Model-free Reinforcement Learning in Infinite-horizon Average-reward Markov Decision Processes

2019-10-15 · ICML 2020 1 · Chen-Yu Wei, Mehdi Jafarnia-Jahromi, Haipeng Luo, Hiteshi Sharma 외

Model-free reinforcement learning is known to be memory and computation efficient and more amendable to large scale problems. In this paper, two model-free algorithms are introduced for learning infinite-horizon average-…

Multi-Armed Bandits reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Examining average and discounted reward optimality criteria in reinforcement learning

2021-07-03 · Vektor Dewanto, Marcus Gallagher

In reinforcement learning (RL), the goal is to obtain an optimal policy, for which the optimality criterion is fundamentally important. Two major optimality criteria are average and discounted rewards. While the latter i…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Average-reward model-free reinforcement learning: a systematic review and literature mapping

2020-10-18 · Vektor Dewanto, George Dunn, Ali Eshragh, Marcus Gallagher 외

Reinforcement learning is important part of artificial intelligence. In this paper, we review model-free reinforcement learning that utilizes the average reward optimality criterion in the infinite horizon setting. Motiv…

Decision Making reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1

Average-Reward Maximum Entropy Reinforcement Learning for Underactuated Double Pendulum Tasks

2024-09-13 · Jean Seong Bjorn Choe, Bumkyu Choi, Jong-Kook Kim

This report presents a solution for the swing-up and stabilisation tasks of the acrobot and the pendubot, developed for the AI Olympics competition at IROS 2024. Our approach employs the Average-Reward Entropy Advantage …

Acrobot Reinforcement Learning (RL)

A Model-free Learning Algorithm for Infinite-horizon Average-reward MDPs with Near-optimal Regret

2020-06-08 · Mehdi Jafarnia-Jahromi, Chen-Yu Wei, Rahul Jain, Haipeng Luo

Recently, model-free reinforcement learning has attracted research attention due to its simplicity, memory and computation efficiency, and the flexibility to combine with function approximation. In this paper, we propose…

Q-Learning reinforcement-learning Reinforcement Learning (RL)