Langevin Soft Actor-Critic: Efficient Exploration through Uncertainty-Driven Critic Learning
Existing actor-critic algorithms, which are popular for continuous control reinforcement learning (RL) tasks, suffer from poor sample efficiency due to lack of principled exploration mechanism within them. Motivated by the success of Thompson sampling for efficient exploration in RL, we propose a novel model-free RL algorithm, Langevin Soft Actor Critic (LSAC), which prioritizes enhancing critic learning through uncertainty estimation over policy optimization. LSAC employs three key innovations: approximate Thompson sampling through distributional Langevin Monte Carlo (LMC) based $Q$ updates, parallel tempering for exploring multiple modes of the posterior of the $Q$ function, and diffusion synthesized state-action samples regularized with $Q$ action gradients. Our extensive experiments demonstrate that LSAC outperforms or matches the performance of mainstream model-free RL algorithms for continuous control tasks. Notably, LSAC marks the first successful application of an LMC based Thompson sampling in continuous control tasks with continuous action spaces.
Code (1)
Tasks
continuous-controlContinuous ControlEfficient ExplorationReinforcement Learning (RL)Thompson SamplingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Safe Langevin Soft Actor Critic
Balancing reward and safety in constrained reinforcement learning remains challenging due to poor generalization from sharp value minima and inadequate handling of heavy-tailed risk distribution. We introduce Safe Langev…
Reinforcement LearningReal-Time Generative Policy via Langevin-Guided Flow Matching for Autonomous Driving
Reinforcement learning (RL) is a fundamental methodology in autonomous driving systems, where generative policies exhibit considerable potential by leveraging their ability to model complex distributions to enhance explo…
Reinforcement LearningAutonomous DrivingWasserstein Barycenter Soft Actor-Critic
Deep off-policy actor-critic algorithms have emerged as the leading framework for reinforcement learning in continuous control domains. However, most of these algorithms suffer from poor sample efficiency, especially in …
continuous-controlContinuous ControlMuJoCoDirect Soft-Policy Sampling via Langevin Dynamics
Soft policies in reinforcement learning define policies as Boltzmann distributions over state-action value functions, providing a principled mechanism for balancing exploration and exploitation. However, realizing such s…
Reinforcement LearningOpenAI GymOn Generalization and Distributional Update for Mimicking Observations with Adequate Exploration
This paper tackles the efficiency and stability issues in learning from observations (LfO). We commence by investigating how reward functions and policies generalize in LfO. Subsequently, the built-in reinforcement learn…
Reinforcement Learning (RL)