SPP-RL: State Planning Policy Reinforcement Learning
We introduce an algorithm for reinforcement learning, in which the actor plans for the next state provided the current state. To communicate the actor output to the environment we incorporate an inverse dynamics control model and train it using supervised learning. We train the RL agent using off-policy state-of-the-art reinforcement learning algorithms: DDPG, TD3, and SAC. To guarantee that the target states are physically relevant, the overall learning procedure is formulated as a constrained optimization problem, solved via the classical Lagrangian optimization method. We benchmark the state planning RL approach using a varied set of continuous environments, including standard MuJoCo tasks, safety-gym level 0 environments, and AntPush. In SPP approach the optimal policy is being searched for in the space of state-state mappings, a considerably larger space than the traditional space of state-action mappings. We report that quite surprisingly SPP implementations attain superior performance to vanilla state-of-the-art off-policy RL algorithms in the tested environments.
Code (0)
등록된 구현이 없습니다.
Tasks
MuJoCoreinforcement-learningReinforcement LearningReinforcement Learning (RL)Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Integration of Reinforcement Learning Based Behavior Planning With Sampling Based Motion Planning for Automated Driving
Reinforcement learning has received high research interest for developing planning approaches in automated driving. Most prior works consider the end-to-end planning task that yields direct control commands and rarely de…
Deep Reinforcement LearningMotion Planningreinforcement-learningReinforcement LearningImproving the Exploration of Deep Reinforcement Learning in Continuous Domains using Planning for Policy Search
Local policy search is performed by most Deep Reinforcement Learning (D-RL) methods, which increases the risk of getting trapped in a local minimum. Furthermore, the availability of a simulation model is not fully exploi…
Deep Reinforcement LearningModel-based Reinforcement Learningreinforcement-learningReinforcement Learning+1Distilling a Hierarchical Policy for Planning and Control via Representation and Reinforcement Learning
We present a hierarchical planning and control framework that enables an agent to perform various tasks and adapt to a new task flexibly. Rather than learning an individual policy for each particular task, the proposed f…
reinforcement-learningReinforcement Learning (RL)Policy Tree Network
Decision-time planning policies with implicit dynamics models have been shown to work in discrete action spaces with Q learning. However, decision-time planning with implicit dynamics models in continuous action space ha…
Model-based Reinforcement LearningMuJoCoPolicy Gradient MethodsQ-Learning+3Spatially and Seamlessly Hierarchical Reinforcement Learning for State Space and Policy space in Autonomous Driving
Despite advances in hierarchical reinforcement learning, its applications to path planning in autonomous driving on highways are challenging. One reason is that conventional hierarchical reinforcement learning approaches…
Autonomous DrivingHierarchical Reinforcement Learningreinforcement-learningReinforcement Learning+1