Convergent Reinforcement Learning with Function Approximation: A Bilevel Optimization Perspective

2018-09-27 · Zhuoran Yang, Zuyue Fu, Kaiqing Zhang, Zhaoran Wang

We study reinforcement learning algorithms with nonlinear function approximation in the online setting. By formulating both the problems of value function estimation and policy learning as bilevel optimization problems, we propose online Q-learning and actor-critic algorithms for these two problems respectively. Our algorithms are gradient-based methods and thus are computationally efficient. Moreover, by approximating the iterates using differential equations, we establish convergence guarantees for the proposed algorithms. Thorough numerical experiments are conducted to back up our theory.

📄 PDF Abstract BibTeX

Code (0)

등록된 구현이 없습니다.

Tasks

Bilevel OptimizationQ-Learningreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Value Function Based Difference-of-Convex Algorithm for Bilevel Hyperparameter Selection Problems

2022-06-13 · Lucy Gao, Jane J. Ye, Haian Yin, Shangzhi Zeng 외

Gradient-based optimization methods for hyperparameter tuning guarantee theoretical convergence to stationary solutions when for fixed upper-level variable values, the lower level of the bilevel program is strongly conve…

Convergent Policy Optimization for Safe Reinforcement Learning

2019-10-26 · NeurIPS 2019 12 · Ming Yu, Zhuoran Yang, Mladen Kolar, Zhaoran Wang

We study the safe reinforcement learning problem with nonlinear function approximation, where policy optimization is formulated as a constrained optimization problem with both the objective and the constraint being nonco…

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1

Learning Theory for Kernel Bilevel Optimization

2025-02-12 · Fares El Khoury, Edouard Pauwels, Samuel Vaiter, Michael Arbel

Bilevel optimization has emerged as a technique for addressing a wide range of machine learning problems that involve an outer objective implicitly determined by the minimizer of an inner problem. In this paper, we inves…

Bilevel Optimization Learning Theory

Convergent Reinforcement Learning Algorithms for Stochastic Shortest Path Problem

2025-08-19 · Soumyajit Guin, Shalabh Bhatnagar arxiv

In this paper we propose two algorithms in the tabular setting and an algorithm for the function approximation setting for the Stochastic Shortest Path (SSP) problem. SSP problems form an important class of problems in R…

Reinforcement Learning

Efficient Gradient Approximation Method for Constrained Bilevel Optimization

2023-02-03 · Siyuan Xu, Minghui Zhu

Bilevel optimization has been developed for many machine learning tasks with large-scale and high-dimensional data. This paper considers a constrained bilevel optimization problem, where the lower-level optimization prob…

Bilevel Optimization Hyperparameter Optimization Meta-Learning