Versatile Offline Imitation from Observations and Examples via Regularized State-Occupancy Matching
We propose State Matching Offline DIstribution Correction Estimation (SMODICE), a novel and versatile regression-based offline imitation learning (IL) algorithm derived via state-occupancy matching. We show that the SMODICE objective admits a simple optimization procedure through an application of Fenchel duality and an analytic solution in tabular MDPs. Without requiring access to expert actions, SMODICE can be effectively applied to three offline IL settings: (i) imitation from observations (IfO), (ii) IfO with dynamics or morphologically mismatched expert, and (iii) example-based reinforcement learning, which we show can be formulated as a state-occupancy matching problem. We extensively evaluate SMODICE on both gridworld environments as well as on high-dimensional offline benchmarks. Our results demonstrate that SMODICE is effective for all three problem settings and significantly outperforms prior state-of-art.
Code (2)
Tasks
Imitation LearningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
A Simple Solution for Offline Imitation from Observations and Examples with Possibly Incomplete Trajectories
Offline imitation from observations aims to solve MDPs where only task-specific expert states and task-agnostic non-expert state-action pairs are available. Offline imitation is useful in real-world scenarios where arbit…
A prior regularized full waveform inversion using generative diffusion models
Full waveform inversion (FWI) has the potential to provide high-resolution subsurface model estimations. However, due to limitations in observation, e.g., regional noise, limited shots or receivers, and band-limited data…
MAHALO: Unifying Offline Reinforcement Learning and Imitation Learning from Observations
We study a new paradigm for sequential decision making, called offline policy learning from observations (PLfO). Offline PLfO aims to learn policies using datasets with substandard qualities: 1) only a subset of trajecto…
Decision MakingImitation LearningOffline RLreinforcement-learning+3Nonparametric Online Learning Using Lipschitz Regularized Deep Neural Networks
Deep neural networks are considered to be state of the art models in many offline machine learning tasks. However, their performance and generalization abilities in online learning tasks are much less understood. Therefo…
BIG-bench Machine LearningBRAC+: Improved Behavior Regularized Actor Critic for Offline Reinforcement Learning
Online interactions with the environment to collect data samples for training a Reinforcement Learning (RL) agent is not always feasible due to economic and safety concerns. The goal of Offline Reinforcement Learning is …
Offline RLreinforcement-learningReinforcement LearningReinforcement Learning (RL)