On-line Policy Improvement using Monte-Carlo Search

2025-01-09 · Gerald Tesauro, Gregory R. Galperin

We present a Monte-Carlo simulation algorithm for real-time policy improvement of an adaptive controller. In the Monte-Carlo simulation, the long-term expected reward of each possible action is statistically measured, using the initial policy to make decisions in each step of the simulation. The action maximizing the measured expected reward is then taken, resulting in an improved policy. Our algorithm is easily parallelizable and has been implemented on the IBM SP1 and SP2 parallel-RISC supercomputers. We have obtained promising initial results in applying this algorithm to the domain of backgammon. Results are reported for a wide variety of initial policies, ranging from a random policy to TD-Gammon, an extremely strong multi-layer neural network. In each case, the Monte-Carlo algorithm gives a substantial reduction, by as much as a factor of 5 or more, in the error rate of the base players. The algorithm is also potentially useful in many other adaptive control applications in which it is possible to simulate the environment.

📄 PDF Abstract BibTeX arXiv:2501.05407

Code (0)

등록된 구현이 없습니다.

Methods 이 논문이 사용한 방법론

Accumulating Eligibility Trace An Accumulating Eligibility Trace is a type of eligibility trace where the trace increments in an accumulative way. For…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

TD Lambda TD_INLINE_MATH_1 is a generalisation of TD_INLINE_MATH_2 reinforcement learning algorithms, but it employs an [eligibility…

Feedforward Network A Feedforward Network, or a Multilayer Perceptron (MLP), is a neural network with solely densely connected layers. This is the classic neural network architecture of the…

TD-Gammon TD-Gammon is a game-learning architecture for playing backgammon. It involves the use of a $TD\left(\lambda\right)$ learning algorithm and a feedforward neural…

BASE 설명 없음

Is Monte Carlo Tree Search Just Every-Visit Monte Carlo Control?

2026-08-28 · Xianyi Wu arxiv

Monte Carlo Tree Search (MCTS) and every-visit Monte Carlo (MC) control are usually presented as different methods. MCTS is described in the language of search (selection, expansion, simulation, and backup), whereas MC c…

Reinforcement Learning

PMCTS: Particle Monte Carlo Tree Search for Principled Parallelized Inference Time Scaling

2026-05-09 · Yaniv Oren, Viliam Vadocz, Joery A. de Vries, Wendelin Böhmer 외 arxiv

Monte Carlo Tree Search (MCTS) is a widely used approach for policy improvement through search with increasing popularity for real world applications. Due to the sequential and deterministic nature of its search, runtime…

Monte-Carlo Graph Search for AlphaZero

2020-12-20 · Johannes Czech, Patrick Korus, Kristian Kersting

The AlphaZero algorithm has been successfully applied in a range of discrete domains, most notably board games. It utilizes a neural network, that learns a value and policy function to guide the exploration in a Monte-Ca…

Board Games

Bayesian Optimized Monte Carlo Planning

2020-10-07 · John Mern, Anil Yildiz, Zachary Sunberg, Tapan Mukerji 외

Online solvers for partially observable Markov decision processes have difficulty scaling to problems with large action spaces. Monte Carlo tree search with progressive widening attempts to improve scaling by sampling fr…

Bayesian Optimization

Policy Learning and Evaluation with Randomized Quasi-Monte Carlo

2022-02-16 · Sebastien M. R. Arnold, Pierre L'Ecuyer, Liyu Chen, Yi-fan Chen 외

Reinforcement learning constantly deals with hard integrals, for example when computing expectations in policy evaluation and policy iteration. These integrals are rarely analytically solvable and typically estimated wit…

continuous-control Continuous Control Policy Gradient Methods Reinforcement Learning (RL)