Multi-timescale Nexting in a Reinforcement Learning Robot

2011-12-06 · Joseph Modayil, Adam White, Richard S. Sutton

The term "nexting" has been used by psychologists to refer to the propensity of people and many other animals to continually predict what will happen next in an immediate, local, and personal sense. The ability to "next" constitutes a basic kind of awareness and knowledge of one's environment. In this paper we present results with a robot that learns to next in real time, predicting thousands of features of the world's state, including all sensory inputs, at timescales from 0.1 to 8 seconds. This was achieved by treating each state feature as a reward-like target and applying temporal-difference methods to learn a corresponding value function with a discount rate corresponding to the timescale. We show that two thousand predictions, each dependent on six thousand state features, can be learned and updated online at better than 10Hz on a laptop computer, using the standard TD(lambda) algorithm with linear function approximation. We show that this approach is efficient enough to be practical, with most of the learning complete within 30 minutes. We also show that a single tile-coded feature representation suffices to accurately predict many different signals at a significant range of timescales. Finally, we show that the accuracy of our learned predictions compares favorably with the optimal off-line solution.

📄 PDF Abstract BibTeX arXiv:1112.1133

Code (0)

등록된 구현이 없습니다.

Tasks

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

A Comparison of Prediction Algorithms and Nexting for Short Term Weather Forecasts

2019-03-18 · Michael Koller, Johannes Feldmaier, Klaus Diepold

This report first provides a brief overview of a number of supervised learning algorithms for regression tasks. Among those are neural networks, regression trees, and the recently introduced Nexting. Nexting has been pre…

regression Reinforcement Learning Reinforcement Learning (RL)

Gamma-Nets: Generalizing Value Estimation over Timescale

2019-11-18 · Craig Sherstan, Shibhansh Dohare, James Macglashan, Johannes Günther 외

We present $\Gamma$-nets, a method for generalizing value function estimation over timescale. By using the timescale as one of the estimator's inputs we can estimate value for arbitrary timescales. As a result, the predi…

Deep Reinforcement Learning Lifelong learning Reinforcement Learning Representation Learning

A Novel Multi-Timescale Stability-Preserving Hierarchical Reinforcement Learning Controller Framework for Adaptive Control in High-Dimensional Dynamical Systems

2025-10-25 · Mohammad Ali Labbaf Khaniki, Fateme Taroodi, Benyamin Safizadeh arxiv

Controlling high-dimensional stochastic systems, critical in robotics, autonomous vehicles, and hyperchaotic systems, faces the curse of dimensionality, lacks temporal abstraction, and often fails to ensure stochastic st…

Hierarchical Reinforcement Learning Autonomous Vehicles

Non-Stationary Policy Learning for Multi-Timescale Multi-Agent Reinforcement Learning

2023-07-17 · Patrick Emami, Xiangyu Zhang, David Biagioni, Ahmed S. Zamzam

In multi-timescale multi-agent reinforcement learning (MARL), agents interact across different timescales. In general, policies for time-dependent behaviors, such as those induced by multiple timescales, are non-stationa…

energy management Inductive Bias Management Multi-agent Reinforcement Learning+1

Multi-timescale memory dynamics in a reinforcement learning network with attention-gated memory

2017-12-28 · Marco Martinolli, Wulfram Gerstner, Aditya Gilra

Learning and memory are intertwined in our brain and their relationship is at the core of several recent neural network models. In particular, the Attention-Gated MEmory Tagging model (AuGMEnT) is a reinforcement learnin…