Learning to Reweight Imaginary Transitions for Model-Based Reinforcement Learning

2021-04-09 · Wenzhen Huang, Qiyue Yin, Junge Zhang, Kaiqi Huang

Model-based reinforcement learning (RL) is more sample efficient than model-free RL by using imaginary trajectories generated by the learned dynamics model. When the model is inaccurate or biased, imaginary trajectories may be deleterious for training the action-value and policy functions. To alleviate such problem, this paper proposes to adaptively reweight the imaginary transitions, so as to reduce the negative effects of poorly generated trajectories. More specifically, we evaluate the effect of an imaginary transition by calculating the change of the loss computed on the real samples when we use the transition to train the action-value and policy functions. Based on this evaluation criterion, we construct the idea of reweighting each imaginary transition by a well-designed meta-gradient algorithm. Extensive experimental results demonstrate that our method outperforms state-of-the-art model-based and model-free RL algorithms on multiple tasks. Visualization of our changing weights further validates the necessity of utilizing reweight scheme.

📄 PDF Abstract BibTeX arXiv:2104.04174

Code (0)

등록된 구현이 없습니다.

Tasks

Model-based Reinforcement Learningreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Deep autoregressive density nets vs neural ensembles for model-based offline reinforcement learning

2024-02-05 · Abdelhakim Benechehab, Albert Thomas, Balázs Kégl

We consider the problem of offline reinforcement learning where only a set of system transitions is made available for policy optimization. Following recent advances in the field, we consider a model-based reinforcement …

D4RL Model-based Reinforcement Learning reinforcement-learning Reinforcement Learning

Bridging Imagination and Reality for Model-Based Deep Reinforcement Learning

2020-10-23 · NeurIPS 2020 12 · Guangxiang Zhu, Minghao Zhang, Honglak Lee, Chongjie Zhang

Sample efficiency has been one of the major challenges for deep reinforcement learning. Recently, model-based reinforcement learning has been proposed to address this challenge by performing planning on imaginary traject…

Deep Reinforcement Learning Model-based Reinforcement Learning reinforcement-learning Reinforcement Learning+1

Sample-efficient Deep Reinforcement Learning with Imaginary Rollouts for Human-Robot Interaction

2019-08-15 · Mohammad Thabet, Massimiliano Patacchiola, Angelo Cangelosi

Deep reinforcement learning has proven to be a great success in allowing agents to learn complex tasks. However, its application to actual robots can be prohibitively expensive. Furthermore, the unpredictability of human…

Deep Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

ImagineBench: Evaluating Reinforcement Learning with Large Language Model Rollouts

2025-05-15 · Jing-Cheng Pang, Kaiyuan Li, Yidi Wang, Si-Hang Yang 외

A central challenge in reinforcement learning (RL) is its dependence on extensive real-world interaction data to learn task-specific policies. While recent work demonstrates that large language models (LLMs) can mitigate…

Continual Learning Language Modeling Language Modelling Large Language Model+4

Revisiting Design Choices in Offline Model-Based Reinforcement Learning

2021-10-08 · Cong Lu, Philip J. Ball, Jack Parker-Holder, Michael A. Osborne 외

Offline reinforcement learning enables agents to leverage large pre-collected datasets of environment transitions to learn control policies, circumventing the need for potentially expensive or unsafe online data collecti…

Bayesian Optimization Model-based Reinforcement Learning reinforcement-learning Reinforcement Learning+1