Deep PQR: Solving Inverse Reinforcement Learning using Anchor Actions

2020-07-15 · Sinong Geng, Houssam Nassif, Carlos A. Manzanares, A. Max Reppen, Ronnie Sircar

We propose a reward function estimation framework for inverse reinforcement learning with deep energy-based policies. We name our method PQR, as it sequentially estimates the Policy, the $Q$-function, and the Reward function by deep learning. PQR does not assume that the reward solely depends on the state, instead it allows for a dependency on the choice of action. Moreover, PQR allows for stochastic state transitions. To accomplish this, we assume the existence of one anchor action whose reward is known, typically the action of doing nothing, yielding no reward. We present both estimators and algorithms for the PQR method. When the environment transition is known, we prove that the PQR reward estimator uniquely recovers the true reward. With unknown transitions, we bound the estimation error of PQR. Finally, the performance of PQR is demonstrated by synthetic and real-world datasets.

📄 PDF Abstract BibTeX arXiv:2007.07443

Code (1)

gengsinong/samq pytorch

Tasks

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Inverse Reinforcement Learning with Just Classification and a Few Regressions

2025-09-25 · Lars van der Laan, Nathan Kallus, Aurelien Bibaut arxiv

Inverse reinforcement learning (IRL) aims to infer rewards from observed behavior, but rewards are not identified from the policy alone: many reward--value pairs can rationalize the same actions. Meaningful reward recove…

Reinforcement Learning

Inverting Non-Injective Functions with Twin Neural Network Regression

2026-01-08 · Sebastian J. Wetzel arxiv

Non-injective functions are not globally invertible. However, they can often be restricted to locally injective subdomains where the inversion is well-defined. In many settings a preferred solution can be selected even w…

Inverse Reinforcement Learning with the Average Reward Criterion

2023-05-24 · NeurIPS 2023 11

We study the problem of Inverse Reinforcement Learning (IRL) with an average-reward criterion. The goal is to recover an unknown policy and a reward function when the agent only has samples of states and actions from an …

MuJoCo reinforcement-learning Reinforcement Learning

Cooperative Inverse Reinforcement Learning

2016-06-09 · NeurIPS 2016 12 · Dylan Hadfield-Menell, Anca Dragan, Pieter Abbeel, Stuart Russell

For an autonomous system to be helpful to humans and to pose no unwarranted risks, it needs to align its values with those of the humans in its environment in such a way that its actions contribute to the maximization of…

Active Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Learning Reward Models for Cooperative Trajectory Planning with Inverse Reinforcement Learning and Monte Carlo Tree Search

2022-02-14 · Karl Kurzer, Matthias Bitzer, J. Marius Zöllner

Cooperative trajectory planning methods for automated vehicles can solve traffic scenarios that require a high degree of cooperation between traffic participants. However, for cooperative systems to integrate into human-…

Decision Making reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1