Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learning
Score-based generative models like the diffusion model have been testified to be effective in modeling multi-modal data from image generation to reinforcement learning (RL). However, the inference process of diffusion model can be slow, which hinders its usage in RL with iterative sampling. We propose to apply the consistency model as an efficient yet expressive policy representation, namely consistency policy, with an actor-critic style algorithm for three typical RL settings: offline, offline-to-online and online. For offline RL, we demonstrate the expressiveness of generative models as policies from multi-modal data. For offline-to-online RL, the consistency policy is shown to be more computational efficient than diffusion policy, with a comparable performance. For online RL, the consistency policy demonstrates significant speedup and even higher average performances than the diffusion policy.
Code (1)
Tasks
Image GenerationOffline RLReinforcement Learning (RL)Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Implicit Policy for Reinforcement Learning
We introduce Implicit Policy, a general class of expressive policies that can flexibly represent complex action distributions in reinforcement learning, with efficient algorithms to compute entropy regularized policy gra…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Efficient Inference for Inverse Reinforcement Learning and Dynamic Discrete Choice Models
In many sequential decision-making problems, researchers observe actions but not the rewards that drive behavior, yet still wish to evaluate and compare counterfactual policies. Inverse reinforcement learning (IRL) and d…
Reinforcement LearningRSPO: Reward-Swap Policy Optimization for Multi-Turn LLM Agents
Reinforcement learning holds significant potential for training large language models (LLMs) to handle multi-turn interactive tasks. However, in long-horizon, multi-turn tasks characterized by sparse outcome rewards, dir…
Reinforcement LearningReinforcement Learning for Economic Policy: A New Frontier?
Agent-based computational economics is a field with a rich academic history, yet one which has struggled to enter mainstream policy design toolboxes, plagued by the challenges associated with representing a complex and d…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency Policy
Vision-Language-Action (VLA) models have shown substantial potential in real-world robotic manipulation. However, fine-tuning these models through supervised learning struggles to achieve robust performance due to limite…
Q-LearningSafe ExplorationVision-Language-Action