Understanding Reward Ambiguity Through Optimal Transport Theory in Inverse Reinforcement Learning
In inverse reinforcement learning (IRL), the central objective is to infer underlying reward functions from observed expert behaviors in a way that not only explains the given data but also generalizes to unseen scenarios. This ensures robustness against reward ambiguity where multiple reward functions can equally explain the same expert behaviors. While significant efforts have been made in addressing this issue, current methods often face challenges with high-dimensional problems and lack a geometric foundation. This paper harnesses the optimal transport (OT) theory to provide a fresh perspective on these challenges. By utilizing the Wasserstein distance from OT, we establish a geometric framework that allows for quantifying reward ambiguity and identifying a central representation or centroid of reward functions. These insights pave the way for robust IRL methodologies anchored in geometric interpretations, offering a structured approach to tackle reward ambiguity in high-dimensional settings.
Code (0)
등록된 구현이 없습니다.
Tasks
reinforcement-learningSimilar Papers 제목 키워드 기반
Designing Ambiguity Sets for Distributionally Robust Optimization Using Structural Causal Optimal Transport
Distributionally robust optimization tackles out-of-sample issues like overfitting and distribution shifts by adopting an adversarial approach over a range of possible data distributions, known as the ambiguity set. To b…
Align Your Intents: Offline Imitation Learning via Optimal Transport
Offline Reinforcement Learning (RL) addresses the problem of sequential decision-making by learning optimal policy through pre-collected data, without interacting with the environment. As yet, it has remained somewhat im…
D4RLDecision MakingImitation LearningOffline RL+2Semi-Discrete Optimal Transport: Hardness, Regularization and Numerical Solution
Semi-discrete optimal transport problems, which evaluate the Wasserstein distance between a discrete and a generic (possibly non-discrete) probability measure, are believed to be computationally hard. Even though such pr…
Discrete Choice ModelsOptimal Transport for Offline Imitation Learning
With the advent of large datasets, offline reinforcement learning (RL) is a promising framework for learning good decision-making policies without the need to interact with the real environment. However, offline RL requi…
D4RLDecision MakingImitation LearningOffline RL+1Non-Rectangular Average-Reward Robust MDPs: Optimal Policies and Their Transient Values
We study non-rectangular robust Markov decision processes under the average-reward criterion, where the ambiguity set couples transition probabilities across states and the adversary commits to a stationary kernel for th…
Reinforcement Learning