Gap-Increasing Policy Evaluation for Efficient and Noise-Tolerant Reinforcement Learning

2019-06-18 · Tadashi Kozuno, Dongqi Han, Kenji Doya

In real-world applications of reinforcement learning (RL), noise from inherent stochasticity of environments is inevitable. However, current policy evaluation algorithms, which plays a key role in many RL algorithms, are either prone to noise or inefficient. To solve this issue, we introduce a novel policy evaluation algorithm, which we call Gap-increasing RetrAce Policy Evaluation (GRAPE). It leverages two recent ideas: (1) gap-increasing value update operators in advantage learning for noise-tolerance and (2) off-policy eligibility trace in Retrace algorithm for efficient learning. We provide detailed theoretical analysis of the new algorithm that shows its efficiency and noise-tolerance inherited from Retrace and advantage learning. Furthermore, our analysis shows that GRAPE's learning is significantly efficient than that of a simple learning-rate-based approach while keeping the same level of noise-tolerance. We applied GRAPE to control problems and obtained experimental results supporting our theoretical analysis.

📄 PDF Abstract BibTeX arXiv:1906.07586

Code (0)

등록된 구현이 없습니다.

Tasks

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Retrace Retrace is an off-policy Q-value estimation algorithm which has guaranteed convergence for a target and behaviour policy $\left(\pi, \beta\right)$. With off-policy rollout for…

Eligibility Trace An Eligibility Trace is a memory vector $\textbf{z}\_{t} \in \mathbb{R}^{d}$ that parallels the long-term weight vector $\textbf{w}\_{t} \in \mathbb{R}^{d}$. The idea is that…

Comparison of Model Predictive and Reinforcement Learning Methods for Fault Tolerant Control

2020-08-10 · Ibrahim Ahmed, Hamed Khorasgani, Gautam Biswas

A desirable property in fault-tolerant controllers is adaptability to system changes as they evolve during systems operations. An adaptive controller does not require optimal control policies to be enumerated for possibl…

Hierarchical Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Fault-Tolerant Control of Degrading Systems with On-Policy Reinforcement Learning

2020-08-10 · Ibrahim Ahmed, Marcos Quiñones-Grueiro, Gautam Biswas

We propose a novel adaptive reinforcement learning control approach for fault tolerant control of degrading systems that is not preceded by a fault detection and diagnosis step. Therefore, \textit{a priori} knowledge of …

Fault Detection reinforcement-learning Reinforcement Learning (RL)

Reinforcement Learning Decoders for Fault-Tolerant Quantum Computation

2018-10-16 · Ryan Sweke, Markus S. Kesselring, Evert P. L. van Nieuwenburg, Jens Eisert

Topological error correcting codes, and particularly the surface code, currently provide the most feasible roadmap towards large-scale fault-tolerant quantum computation. As such, obtaining fast and flexible decoding alg…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Synthetic Aperture Radar Image Change Detection via Layer Attention-Based Noise-Tolerant Network

2022-08-09 · Desen Meng, Feng Gao, Junyu Dong, Qian Du 외

Recently, change detection methods for synthetic aperture radar (SAR) images based on convolutional neural networks (CNN) have gained increasing research attention. However, existing CNN-based methods neglect the interac…

Change Detection

Rethinking the Design of Reinforcement Learning-Based Deep Research Agents

2025-10-17 · Yi Wan, Jiuqi Wang, Liam Li, Jinsong Liu 외 arxiv

Large language models (LLMs) augmented with external tools are increasingly deployed as deep research agents that gather, reason over, and synthesize web information to answer complex queries. Although recent open-source…

Reinforcement Learning