Group Distributionally Robust Reinforcement Learning with Hierarchical Latent Variables
One key challenge for multi-task Reinforcement learning (RL) in practice is the absence of task indicators. Robust RL has been applied to deal with task ambiguity, but may result in over-conservative policies. To balance the worst-case (robustness) and average performance, we propose Group Distributionally Robust Markov Decision Process (GDR-MDP), a flexible hierarchical MDP formulation that encodes task groups via a latent mixture model. GDR-MDP identifies the optimal policy that maximizes the expected return under the worst-possible qualified belief over task groups within an ambiguity set. We rigorously show that GDR-MDP's hierarchical structure improves distributional robustness by adding regularization to the worst possible outcomes. We then develop deep RL algorithms for GDR-MDP for both value-based and policy-based RL methods. Extensive experiments on Box2D control tasks, MuJoCo benchmarks, and Google football platforms show that our algorithms outperform classic robust training algorithms across diverse environments in terms of robustness under belief uncertainties. Demos are available on our project page (\url{https://sites.google.com/view/gdr-rl/home}).
Code (0)
등록된 구현이 없습니다.
Tasks
MuJoCoreinforcement-learningReinforcement LearningReinforcement Learning (RL)Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Latent Space Policies for Hierarchical Reinforcement Learning
We address the problem of learning hierarchical deep neural network policies for reinforcement learning. In contrast to methods that explicitly restrict or cripple lower layers of a hierarchy to force them to use higher-…
Hierarchical Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)Towards Identifiability of Hierarchical Temporal Causal Representation Learning
Modeling hierarchical latent dynamics behind time series data is critical for capturing temporal dependencies across multiple levels of abstraction in real-world tasks. However, existing temporal causal representation le…
Representation LearningImproving Unsupervised Hierarchical Representation with Reinforcement Learning
Learning representations to capture the very fundamental understanding of the world is a key challenge in machine learning. The hierarchical structure of explanatory factors hidden in data is such a general represent…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Hierarchical Bayesian image analysis: from low-level modeling to robust supervised learning
Within a supervised classification framework, labeled data are used to learn classifier parameters. Prior to that, it is generally required to perform dimensionality reduction via feature extraction. These preprocessing …
ClassificationClusteringDimensionality ReductionGeneral ClassificationBayesian Unsupervised Disentanglement of Anatomy and Geometry for Deep Groupwise Image Registration
This article presents a general Bayesian learning framework for multi-modal groupwise image registration. The method builds on probabilistic modelling of the image generative process, where the underlying common anatomy …
AnatomyBayesian InferenceDisentanglementImage Registration