An Online Prediction Algorithm for Reinforcement Learning with Linear Function Approximation using Cross Entropy Method

2018-06-15 · Ajin George Joseph, Shalabh Bhatnagar

In this paper, we provide two new stable online algorithms for the problem of prediction in reinforcement learning, \emph{i.e.}, estimating the value function of a model-free Markov reward process using the linear function approximation architecture and with memory and computation costs scaling quadratically in the size of the feature set. The algorithms employ the multi-timescale stochastic approximation variant of the very popular cross entropy (CE) optimization method which is a model based search method to find the global optimum of a real-valued function. A proof of convergence of the algorithms using the ODE method is provided. We supplement our theoretical results with experimental comparisons. The algorithms achieve good performance fairly consistently on many RL benchmark problems with regards to computational efficiency, accuracy and stability.

📄 PDF Abstract BibTeX arXiv:1806.06720

Code (0)

등록된 구현이 없습니다.

Tasks

Computational Efficiency Reinforcement Learning Reinforcement Learning (RL)

Adapting the Function Approximation Architecture in Online Reinforcement Learning

2021-06-17 · John D. Martin, Joseph Modayil

The performance of a reinforcement learning (RL) system depends on the computational architecture used to approximate a value function. Deep learning methods provide both optimization techniques and architectures for app…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Nonlinear Inverse Reinforcement Learning with Gaussian Processes

2011-12-01 · NeurIPS 2011 12 · Sergey Levine, Zoran Popovic, Vladlen Koltun

We present a probabilistic algorithm for nonlinear inverse reinforcement learning. The goal of inverse reinforcement learning is to learn the reward function in a Markov decision process from expert demonstrations. While…

Gaussian Processes reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Reinforcement Learning with Function Approximation: From Linear to Nonlinear

2023-02-20 · Jihao Long, Jiequn Han

Function approximation has been an indispensable component in modern reinforcement learning algorithms designed to tackle problems with large state spaces in high dimensions. This paper reviews recent results on error an…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

A Generalized Projected Bellman Error for Off-policy Value Estimation in Reinforcement Learning

2021-04-28 · Andrew Patterson, Adam White, Martha White

Many reinforcement learning algorithms rely on value estimation, however, the most widely used algorithms -- namely temporal difference algorithms -- can diverge under both off-policy sampling and nonlinear function appr…

reinforcement-learning Reinforcement Learning (RL)

Multi-objective Reinforcement Learning with Nonlinear Preferences: Provable Approximation for Maximizing Expected Scalarized Return

2023-11-05 · Nianli Peng, Muhang Tian, Brandon Fain

We study multi-objective reinforcement learning with nonlinear preferences over trajectories. That is, we maximize the expected value of a nonlinear function over accumulated rewards (expected scalarized return or ESR) i…

Fairness Multi-Objective Reinforcement Learning reinforcement-learning