PAC-Bayesian Soft Actor-Critic Learning
Actor-critic algorithms address the dual goals of reinforcement learning (RL), policy evaluation and improvement via two separate function approximators. The practicality of this approach comes at the expense of training instability, caused mainly by the destructive effect of the approximation errors of the critic on the actor. We tackle this bottleneck by employing an existing Probably Approximately Correct (PAC) Bayesian bound for the first time as the critic training objective of the Soft Actor-Critic (SAC) algorithm. We further demonstrate that online learning performance improves significantly when a stochastic actor explores multiple futures by critic-guided random search. We observe our resulting algorithm to compare favorably against the state-of-the-art SAC implementation on multiple classical control and locomotion tasks in terms of both sample efficiency and regret.
Code (1)
Tasks
Reinforcement Learning (RL)Similar Papers 제목 키워드 기반
Bayesian Soft Actor-Critic: A Directed Acyclic Strategy Graph Based Deep Reinforcement Learning
Adopting reasonable strategies is challenging but crucial for an intelligent agent with limited resources working in hazardous, unstructured, and dynamic environments to improve the system's utility, decrease the overall…
continuous-controlContinuous ControlDeep Reinforcement LearningOpenAI Gym+2A Strategy-Oriented Bayesian Soft Actor-Critic Model
Adopting reasonable strategies is challenging but crucial for an intelligent agent with limited resources working in hazardous, unstructured, and dynamic environments to improve the system's utility, decrease the overall…
continuous-controlContinuous ControlMuJoCoOpenAI GymHyper-parameter optimization based on soft actor critic and hierarchical mixture regularization
Hyper-parameter optimization is a crucial problem in machine learning as it aims to achieve the state-of-the-art performance in any model. Great efforts have been made in this field, such as random search, grid search, B…
Bayesian Optimizationreinforcement-learningReinforcement Learning (RL)Deep Exploration with PAC-Bayes
Reinforcement learning for continuous control under delayed rewards is an under-explored problem despite its significance in real life. Many complex skills build on intermediate ones as prerequisites. For instance, a hum…
continuous-controlContinuous ControlDecision Makingreinforcement-learning+1Actor Loss of Soft Actor Critic Explained
This technical report is devoted to explaining how the actor loss of soft actor critic is obtained, as well as the associated gradient estimate. It gives the necessary mathematical background to derive all the presented …