Difference of Convex Functions Programming for Reinforcement Learning

2014-12-01 · NeurIPS 2014 12 · Bilal Piot, Matthieu Geist, Olivier Pietquin

Large Markov Decision Processes (MDPs) are usually solved using Approximate Dynamic Programming (ADP) methods such as Approximate Value Iteration (AVI) or Approximate Policy Iteration (API). The main contribution of this paper is to show that, alternatively, the optimal state-action value function can be estimated using Difference of Convex functions (DC) Programming. To do so, we study the minimization of a norm of the Optimal Bellman Residual (OBR) $T^*Q-Q$, where $T^*$ is the so-called optimal Bellman operator. Controlling this residual allows controlling the distance to the optimal action-value function, and we show that minimizing an empirical norm of the OBR is consistant in the Vapnik sense. Finally, we frame this optimization problem as a DC program. That allows envisioning using the large related literature on DC Programming to address the Reinforcement Leaning (RL) problem.

📄 PDF Abstract BibTeX

Code (0)

등록된 구현이 없습니다.

Tasks

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Difference of Convex Functions Programming Applied to Control with Expert Data

2016-06-03 · Bilal Piot, Matthieu Geist, Olivier Pietquin

This paper reports applications of Difference of Convex functions (DC) programming to Learning from Demonstrations (LfD) and Reinforcement Learning (RL) with expert data. This is made possible because the norm of the Opt…

General Classification reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Convex Programs and Lyapunov Functions for Reinforcement Learning: A Unified Perspective on the Analysis of Value-Based Methods

2022-02-14 · Xingang Guo, Bin Hu

Value-based methods play a fundamental role in Markov decision processes (MDPs) and reinforcement learning (RL). In this paper, we present a unified control-theoretic framework for analyzing valued-based methods such as …

Reinforcement Learning (RL)

Piecewise Linear Regression via a Difference of Convex Functions

2020-07-05 · ICML 2020 1 · Ali Siahkamari, Aditya Gangrade, Brian Kulis, Venkatesh Saligrama

We present a new piecewise linear regression methodology that utilizes fitting a difference of convex functions (DC functions) to the data. These are functions $f$ that may be represented as the difference $\phi_1 - \phi…

regression

Optimization of stochastic switching buffer network via DC programming

2022-07-18 · Chengyan Zhao, Kazunori Sakurama, Masaki Ogura

This letter deals with the optimization problems of stochastic switching buffer networks, where the switching law is governed by Markov process. The dynamical buffer network is introduced, and its application in modeling…

Learning RBM with a DC programming Approach

2017-09-21 · Vidyadhar Upadhya, P. S. Sastry

By exploiting the property that the RBM log-likelihood function is the difference of convex functions, we formulate a stochastic variant of the difference of convex functions (DC) programming to minimize the negative log…