Cyclic Policy Distillation: Sample-Efficient Sim-to-Real Reinforcement Learning with Domain Randomization

2022-07-29 · Yuki Kadokawa, Lingwei Zhu, Yoshihisa Tsurumine, Takamitsu Matsubara

Deep reinforcement learning with domain randomization learns a control policy in various simulations with randomized physical and sensor model parameters to become transferable to the real world in a zero-shot setting. However, a huge number of samples are often required to learn an effective policy when the range of randomized parameters is extensive due to the instability of policy updates. To alleviate this problem, we propose a sample-efficient method named cyclic policy distillation (CPD). CPD divides the range of randomized parameters into several small sub-domains and assigns a local policy to each one. Then local policies are learned while cyclically transitioning to sub-domains. CPD accelerates learning through knowledge transfer based on expected performance improvements. Finally, all of the learned local policies are distilled into a global policy for sim-to-real transfers. CPD's effectiveness and sample efficiency are demonstrated through simulations with four tasks (Pendulum from OpenAIGym and Pusher, Swimmer, and HalfCheetah from Mujoco), and a real-robot, ball-dispersal task. We published code and videos from our experiments at https://github.com/yuki-kadokawa/cyclic-policy-distillation.

📄 PDF Abstract BibTeX arXiv:2207.14561

Code (1)

yuki-kadokawa/cyclic-policy-distillation 공식 구현 pytorch

Tasks

Deep Reinforcement LearningMuJoCoreinforcement-learning Reinforcement Learning (RL)Transfer Learning

Continual Reinforcement Learning deployed in Real-life using Policy Distillation and Sim2Real Transfer

2019-06-11 · René Traoré, Hugo Caselles-Dupré, Timothée Lesort, Te Sun 외

We focus on the problem of teaching a robot to solve tasks presented sequentially, i.e., in a continual learning scenario. The robot should be able to solve all tasks it has encountered, without forgetting past tasks. We…

Continual Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1

Real-time Policy Distillation in Deep Reinforcement Learning

2019-12-29 · Yuxiang Sun, Pooyan Fazli

Policy distillation in deep reinforcement learning provides an effective way to transfer control policies from a larger network to a smaller untrained network without a significant degradation in performance. However, po…

Deep Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Proximal Policy Distillation

2024-07-21 · Giacomo Spigler

We introduce Proximal Policy Distillation (PPD), a novel policy distillation method that integrates student-driven distillation and Proximal Policy Optimization (PPO) to increase sample efficiency and to leverage the add…

continuous-control Continuous Control MuJoCo

Reinforcement Learning for Causal Discovery without Acyclicity Constraints

2024-08-24 · Bao Duong, Hung Le, Biwei Huang, Thin Nguyen

Recently, reinforcement learning (RL) has proved a promising alternative for conventional local heuristics in score-based approaches to learning directed acyclic causal graphs (DAGs) from observational data. However, the…

Causal Discovery Efficient Exploration Navigate Policy Gradient Methods+3

Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing

2026-04-02 · Gengsheng Li, Tianyu Yang, Junfeng Fang, Mingyang Song 외 arxiv

Reinforcement learning with verifiable rewards (RLVR) has become a standard paradigm for post-training large language models. While Group Relative Policy Optimization (GRPO) is widely adopted, its coarse credit assignmen…

Reinforcement Learning