Exploiting the Sign of the Advantage Function to Learn Deterministic Policies in Continuous Domains
In the context of learning deterministic policies in continuous domains, we revisit an approach, which was first proposed in Continuous Actor Critic Learning Automaton (CACLA) and later extended in Neural Fitted Actor Critic (NFAC). This approach is based on a policy update different from that of deterministic policy gradient (DPG). Previous work has observed its excellent performance empirically, but a theoretical justification is lacking. To fill this gap, we provide a theoretical explanation to motivate this unorthodox policy update by relating it to another update and making explicit the objective function of the latter. We furthermore discuss in depth the properties of these updates to get a deeper understanding of the overall approach. In addition, we extend it and propose a new trust region algorithm, Penalized NFAC (PeNFAC). Finally, we experimentally demonstrate in several classic control problems that it surpasses the state-of-the-art algorithms to learn deterministic policies.
Code (1)
Similar Papers 제목 키워드 기반
Zeroth-order Deterministic Policy Gradient
Deterministic Policy Gradient (DPG) removes a level of randomness from standard randomized-action Policy Gradient (PG), and demonstrates substantial empirical success for tackling complex dynamic problems involving Marko…
Deep Reinforcement Learning with Discrete Normalized Advantage Functions for Resource Management in Network Slicing
Network slicing promises to provision diversified services with distinct requirements in one infrastructure. Deep reinforcement learning (e.g., deep $\mathcal{Q}$-learning, DQL) is assumed to be an appropriate algorithm …
Deep Reinforcement LearningManagementQ-LearningReinforcement Learning+2The Effect of Multi-step Methods on Overestimation in Deep Reinforcement Learning
Multi-step (also called n-step) methods in reinforcement learning (RL) have been shown to be more efficient than the 1-step method due to faster propagation of the reward signal, both theoretically and empirically, in ta…
Deep Reinforcement Learningreinforcement-learningReinforcement Learning (RL)Exploiting Functional Dependencies in Qualitative Probabilistic Reasoning
Functional dependencies restrict the potential interactions among variables connected in a probabilistic network. This restriction can be exploited in qualitative probabilistic reasoning by introducing deterministic vari…
Exploiting Noise as a Resource for Computation and Learning in Spiking Neural Networks
$\textbf{Formal version available at}$ https://cell.com/patterns/fulltext/S2666-3899(23)00200-3 Networks of spiking neurons underpin the extraordinary information-processing capabilities of the brain and have become pill…