Lagrangian Method for Episodic Learning

2021-09-29 · Huang Bojun

This paper considers the problem of learning optimal value functions for finite-time decision tasks via saddle-point optimization of a nonlinear Lagrangian function that is derived from the $Q$-form Bellman optimality equation. Despite a long history of research on this topic in the literature, previous works on this general approach have been almost exclusively focusing on a linear special case known as the linear programming approach to RL/MDP. Our paper brings new perspectives to this general approach in the following aspects: 1) Inspired by the usually-used linear $V$-form Lagrangian, we proposed a nonlinear $Q$-form Lagrangian function and proved that it enjoys strong duality property in spite of its nonlinearity. The Lagrangian duality property immediately leads to a new imitation learning algorithm, which we applied to Machine Translation and obtained favorable performance on standard MT benchmark. 2) We pointed out a fundamental limit of existing works, which seeks to find minimax-type saddle points of the Lagrangian function. We proved that another class of saddle points, the maximin-type ones, turn out to have better optimality property. 3) In contrast to most previous works, our theory and algorithm are oriented to the undiscounted episode-wise reward, which is practically more relevant than the usually considered discounted-MDP setting, thus have filled a gap between theory and practice on the topic.

📄 PDF Abstract BibTeX

Code (0)

등록된 구현이 없습니다.

Tasks

FormImitation LearningMachine Translation

Provably Efficient Generalized Lagrangian Policy Optimization for Safe Multi-Agent Reinforcement Learning

2023-05-31 · Dongsheng Ding, Xiaohan Wei, Zhuoran Yang, Zhaoran Wang 외

We examine online safe multi-agent reinforcement learning using constrained Markov games in which agents compete by maximizing their expected total rewards under a constraint on expected total utilities. Our focus is con…

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning Safe Reinforcement Learning

A Unifying View of Optimism in Episodic Reinforcement Learning

2020-07-03 · NeurIPS 2020 12 · Gergely Neu, Ciara Pike-Burke

The principle of optimism in the face of uncertainty underpins many theoretically successful reinforcement learning algorithms. In this paper we provide a general framework for designing, analyzing and implementing such …

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Safe Langevin Soft Actor Critic

2026-01-31 · Mahesh Keswani, Samyak Jain, Raunak P. Bhattacharyya arxiv

Balancing reward and safety in constrained reinforcement learning remains challenging due to poor generalization from sharp value minima and inadequate handling of heavy-tailed risk distribution. We introduce Safe Langev…

Reinforcement Learning

Lagrangian Formalism in Biology: I. Standard Lagrangians and their Role in Population Dynamics

2022-03-24 · D. T. Pham, Z. E. Musielak

The Lagrangian formalism is developed for the population dynamics of interacting species that are described by several well-known models. The formalism is based on standard Lagrangians, which represent differences betwee…

A Review of Lagrangian Formalism in Biology: Recent Advances and Perspectives

2024-08-20 · Diana T. Pham, Zdzislaw E. Musielak

The Lagrangian formalism has attracted the attention of mathematicians and physicists for more than 250 years and has played significant roles in establishing modern theoretical physics. The history of the Lagrangian for…