Cautious Actor-Critic
The oscillating performance of off-policy learning and persisting errors in the actor-critic (AC) setting call for algorithms that can conservatively learn to suit the stability-critical applications better. In this paper, we propose a novel off-policy AC algorithm cautious actor-critic (CAC). The name cautious comes from the doubly conservative nature that we exploit the classic policy interpolation from conservative policy iteration for the actor and the entropy-regularization of conservative value iteration for the critic. Our key observation is the entropy-regularized critic facilitates and simplifies the unwieldy interpolated actor update while still ensuring robust policy improvement. We compare CAC to state-of-the-art AC methods on a set of challenging continuous control problems and demonstrate that CAC achieves comparable performance while significantly stabilizes learning.
Code (0)
등록된 구현이 없습니다.
Tasks
continuous-controlContinuous ControlSimilar Papers 제목 키워드 기반
Cautious Calibration in Binary Classification
Being cautious is crucial for enhancing the trustworthiness of machine learning systems integrated into decision-making pipelines. Although calibrated probabilities help in optimal decision-making, perfect calibration re…
Binary ClassificationClassificationDecision MakingCautious Adaptation For Reinforcement Learning in Safety-Critical Settings
Reinforcement learning (RL) in real-world safety-critical target settings like urban driving is hazardous, imperiling the RL agent, other agents, and the environment. To overcome this difficulty, we propose a "safety-cri…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Cautious Belief and Iterated Admissibility
We define notions of cautiousness and cautious belief to provide epistemic conditions for iterated admissibility in finite games. We show that iterated admissibility characterizes the behavioral implications of "cautious…
Anytime Computation of Cautious Consequences in Answer Set Programming
Query answering in Answer Set Programming (ASP) is usually solved by computing (a subset of) the cautious consequences of a logic program. This task is computationally very hard, and there are programs for which computin…
Learning to Be Cautious
A key challenge in the field of reinforcement learning is to develop agents that behave cautiously in novel situations. It is generally impossible to anticipate all situations that an autonomous system may face or what b…
counterfactualSafe Reinforcement LearningSelf-Driving Cars