Bootstrapping the Expressivity with Model-based Planning
We compare the model-free reinforcement learning with the model-based approaches through the lens of the expressive power of neural networks for policies, $Q$-functions, and dynamics. We show, theoretically and empirically, that even for one-dimensional continuous state space, there are many MDPs whose optimal $Q$-functions and policies are much more complex than the dynamics. We hypothesize many real-world MDPs also have a similar property. For these MDPs, model-based planning is a favorable algorithm, because the resulting policies can approximate the optimal policy significantly better than a neural network parameterization can, and model-free or model-based policy optimization rely on policy parameterization. Motivated by the theory, we apply a simple multi-step model-based bootstrapping planner (BOOTS) to bootstrap a weak $Q$-function into a stronger policy. Empirical results show that applying BOOTS on top of model-based or model-free policy optimization algorithms at the test time improves the performance on MuJoCo benchmark tasks.
Code (1)
Tasks
modelMuJoCoSimilar Papers 제목 키워드 기반
Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learning
We identify an implicit under-parameterization phenomenon in value-based deep RL methods that use bootstrapping: when value functions, approximated using deep neural networks, are trained with gradient descent using iter…
Deep Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)On the Expressivity of Neural Networks for Deep Reinforcement Learning
We compare the model-free reinforcement learning with the model-based approaches through the lens of the expressive power of neural networks for policies, $Q$-functions, and dynamics. We show, theoretically and empirical…
Deep Reinforcement LearningMuJoCoreinforcement-learningReinforcement Learning+1Towards High-Level Modelling in Automated Planning
Planning is a fundamental activity, arising frequently in many contexts, from daily tasks to industrial processes. The planning task consists of selecting a sequence of actions to achieve a specified goal from specified …
Geometric Approaches to Increase the Expressivity of Deep Neural Networks for MR Reconstruction
Recently, deep learning approaches have been extensively investigated to reconstruct images from accelerated magnetic resonance image (MRI) acquisition. Although these approaches provide significant performance gain comp…
compressed sensingDecoderDream and Search to Control: Latent Space Planning for Continuous Control
Learning and planning with latent space dynamics has been shown to be useful for sample efficiency in model-based reinforcement learning (MBRL) for discrete and continuous control tasks. In particular, recent work, for d…
continuous-controlContinuous ControlModel-based Reinforcement LearningReinforcement Learning (RL)