Enhancing Online Reinforcement Learning with Meta-Learned Objective from Offline Data

2025-01-13 · Shilong Deng, Zetao Zheng, Hongcai He, Paul Weng, Jie Shao

A major challenge in Reinforcement Learning (RL) is the difficulty of learning an optimal policy from sparse rewards. Prior works enhance online RL with conventional Imitation Learning (IL) via a handcrafted auxiliary objective, at the cost of restricting the RL policy to be sub-optimal when the offline data is generated by a non-expert policy. Instead, to better leverage valuable information in offline data, we develop Generalized Imitation Learning from Demonstration (GILD), which meta-learns an objective that distills knowledge from offline data and instills intrinsic motivation towards the optimal policy. Distinct from prior works that are exclusive to a specific RL algorithm, GILD is a flexible module intended for diverse vanilla off-policy RL algorithms. In addition, GILD introduces no domain-specific hyperparameter and minimal increase in computational cost. In four challenging MuJoCo tasks with sparse rewards, we show that three RL algorithms enhanced with GILD significantly outperform state-of-the-art methods.

📄 PDF Abstract BibTeX arXiv:2501.07346

Code (1)

sldeng1003/gild 공식 구현 pytorch

Tasks

Imitation LearningMuJoCoReinforcement Learning (RL)

Debiasing Meta-Gradient Reinforcement Learning by Learning the Outer Value Function

2022-11-19 · Clément Bonnet, Laurence Midgley, Alexandre Laterre

Meta-gradient Reinforcement Learning (RL) allows agents to self-tune their hyper-parameters in an online fashion during training. In this paper, we identify a bias in the meta-gradient of current meta-gradient RL approac…

reinforcement-learning Reinforcement Learning (RL)

Improving Generalization in Meta Reinforcement Learning using Learned Objectives

2019-10-09 · ICLR 2020 1 · Louis Kirsch, Sjoerd van Steenkiste, Jürgen Schmidhuber

Biological evolution has distilled the experiences of many learners into the general learning algorithms of humans. Our novel meta reinforcement learning algorithm MetaGenRL is inspired by this process. MetaGenRL distill…

Meta Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Surrogate Learning in Meta-Black-Box Optimization: A Preliminary Study

2025-03-23 · Zeyuan Ma, Zhiyang Huang, Jiacheng Chen, Zhiguang Cao 외

Recent Meta-Black-Box Optimization (MetaBBO) approaches have shown possibility of enhancing the optimization performance through learning meta-level policies to dynamically configure low-level optimizers. However, existi…

Kolmogorov-Arnold Networks Reinforcement Learning (RL)

Meta-Gradient Reinforcement Learning with an Objective Discovered Online

2020-07-16 · NeurIPS 2020 12 · Zhongwen Xu, Hado van Hasselt, Matteo Hessel, Junhyuk Oh 외

Deep reinforcement learning includes a broad family of algorithms that parameterise an internal representation, such as a value function or policy, by a deep neural network. Each algorithm optimises its parameters with r…

Deep Reinforcement Learning Q-Learning reinforcement-learning Reinforcement Learning+1

Offline Meta-Reinforcement Learning with Online Self-Supervision

2021-07-08 · Vitchyr H. Pong, Ashvin Nair, Laura Smith, Catherine Huang 외

Meta-reinforcement learning (RL) methods can meta-train policies that adapt to new tasks with orders of magnitude less data than standard RL, but meta-training itself is costly and time-consuming. If we can meta-train on…

Meta Reinforcement Learning Offline RL reinforcement-learning Reinforcement Learning+1