Maximum Causal Entropy Inverse Reinforcement Learning for Mean-Field Games

2024-01-12 · Berkay Anahtarci, Can Deha Kariksiz, Naci Saldi

In this paper, we introduce the maximum casual entropy Inverse Reinforcement Learning (IRL) problem for discrete-time mean-field games (MFGs) under an infinite-horizon discounted-reward optimality criterion. The state space of a typical agent is finite. Our approach begins with a comprehensive review of the maximum entropy IRL problem concerning deterministic and stochastic Markov decision processes (MDPs) in both finite and infinite-horizon scenarios. Subsequently, we formulate the maximum casual entropy IRL problem for MFGs - a non-convex optimization problem with respect to policies. Leveraging the linear programming formulation of MDPs, we restructure this IRL problem into a convex optimization problem and establish a gradient descent algorithm to compute the optimal solution with a rate of convergence. Finally, we present a new algorithm by formulating the MFG problem as a generalized Nash equilibrium problem (GNEP), which is capable of computing the mean-field equilibrium (MFE) for the forward RL problem. This method is employed to produce data for a numerical example. We note that this novel algorithm is also applicable to general MFE computations.

📄 PDF Abstract BibTeX arXiv:2401.06566

Code (0)

등록된 구현이 없습니다.

Tasks

reinforcement-learning Reinforcement Learning

A Primer on Maximum Causal Entropy Inverse Reinforcement Learning

2022-03-22 · Adam Gleave, Sam Toyer

Inverse Reinforcement Learning (IRL) algorithms infer a reward function that explains demonstrations provided by an expert acting in the environment. Maximum Causal Entropy (MCE) IRL is currently the most popular formula…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Generalized Maximum Causal Entropy for Inverse Reinforcement Learning

2019-11-16 · Tien Mai, Kennard Chan, Patrick Jaillet

We consider the problem of learning from demonstrated trajectories with inverse reinforcement learning (IRL). Motivated by a limitation of the classical maximum entropy model in capturing the structure of the network of …

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Maximum Entropy Inverse Reinforcement Learning for Mean-Field Games with Average Reward

2026-06-15 · Şevket Kaan Alkır, Naci Saldı, Berkay Anahtarcı, Can Deha Karıksız arxiv

We study inverse reinforcement learning for discrete-time, infinite-horizon mean-field games (MFGs) under an average-reward criterion. Expert demonstrations are assumed to arise from a stationary mean-field equilibrium u…

Reinforcement Learning

Measuring Goal-Directedness

2024-12-06 · Matt MacDermott, James Fox, Francesco Belardinelli, Tom Everitt

We define maximum entropy goal-directedness (MEG), a formal measure of goal-directedness in causal models and Markov decision processes, and give algorithms for computing it. Measuring goal-directedness is important, as …

Multi-task Maximum Entropy Inverse Reinforcement Learning

2018-05-22 · Adam Gleave, Oliver Habryka

Multi-task Inverse Reinforcement Learning (IRL) is the problem of inferring multiple reward functions from expert demonstrations. Prior work, built on Bayesian IRL, is unable to scale to complex environments due to compu…

Imitation Learning Meta-Learning reinforcement-learning Reinforcement Learning+1