Heuristic-Guided Reinforcement Learning

2021-06-05 · NeurIPS 2021 12 · Ching-An Cheng, Andrey Kolobov, Adith Swaminathan

We provide a framework for accelerating reinforcement learning (RL) algorithms by heuristics constructed from domain knowledge or offline data. Tabula rasa RL algorithms require environment interactions or computation that scales with the horizon of the sequential decision-making task. Using our framework, we show how heuristic-guided RL induces a much shorter-horizon subproblem that provably solves the original task. Our framework can be viewed as a horizon-based regularization for controlling bias and variance in RL under a finite interaction budget. On the theoretical side, we characterize properties of a good heuristic and its impact on RL acceleration. In particular, we introduce the novel concept of an improvable heuristic, a heuristic that allows an RL agent to extrapolate beyond its prior knowledge. On the empirical side, we instantiate our framework to accelerate several state-of-the-art algorithms in simulated robotic control tasks and procedurally generated games. Our framework complements the rich literature on warm-starting RL with expert demonstrations or exploratory datasets, and introduces a principled method for injecting prior knowledge into RL.

📄 PDF Abstract BibTeX arXiv:2106.02757

Code (0)

등록된 구현이 없습니다.

Tasks

Decision Making reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Sequential Decision Making

Learning Heuristics for Template-based CEGIS of Loop Invariants with Reinforcement Learning

2021-07-16 · Minchao Wu, Takeshi Tsukada, Hiroshi Unno, Taro Sekiyama 외

Loop-invariant synthesis is the basis of program verification. Due to the undecidability of the problem in general, a tool for invariant synthesis necessarily uses heuristics. Despite the common belief that the design of…

reinforcement-learning Reinforcement Learning (RL)

SORREL: Suboptimal-Demonstration-Guided Reinforcement Learning for Learning to Branch

2024-12-20 · Shengyu Feng, Yiming Yang

Mixed Integer Linear Program (MILP) solvers are mostly built upon a Branch-and-Bound (B\&B) algorithm, where the efficiency of traditional solvers heavily depends on hand-crafted heuristics for branching. The past few ye…

Imitation Learning reinforcement-learning Reinforcement Learning

Reinforcement-Guided Hyper-Heuristic Hyperparameter Optimization for Fair and Explainable Spiking Neural Network-Based Financial Fraud Detection

2025-08-23 · Sadman Mohammad Nasif, Md Abrar Jahin, M. F. Mridha arxiv

The growing adoption of home banking systems has increased cyberfraud risks, requiring detection models that are accurate, fair, and explainable. While AI methods show promise, they face challenges including computationa…

Hyperparameter Optimization Reinforcement Learning Fraud Detection

Vanishing Bias Heuristic-guided Reinforcement Learning Algorithm

2023-06-17 · Qinru Li, Hao Xiang

Reinforcement Learning has achieved tremendous success in the many Atari games. In this paper we explored with the lunar lander environment and implemented classical methods including Q-Learning, SARSA, MC as well as til…

Atari Games Q-Learning reinforcement-learning Reinforcement Learning

Deep Reinforcement Learning Guided Improvement Heuristic for Job Shop Scheduling

2022-11-20 · Cong Zhang, Zhiguang Cao, Wen Song, Yaoxin Wu 외

Recent studies in using deep reinforcement learning (DRL) to solve Job-shop scheduling problems (JSSP) focus on construction heuristics. However, their performance is still far from optimality, mainly because the underly…

Deep Reinforcement Learning Graph Neural Network Job Shop Scheduling reinforcement-learning+3