Unsupervised Basis Function Adaptation for Reinforcement Learning

2017-03-03 · Edward W. Barker, Charl J. Ras

When using reinforcement learning (RL) algorithms to evaluate a policy it is common, given a large state space, to introduce some form of approximation architecture for the value function (VF). The exact form of this architecture can have a significant effect on the accuracy of the VF estimate, however, and determining a suitable approximation architecture can often be a highly complex task. Consequently there is a large amount of interest in the potential for allowing RL algorithms to adaptively generate approximation architectures. We investigate a method of adapting approximation architectures which uses feedback regarding the frequency with which an agent has visited certain states to guide which areas of the state space to approximate with greater detail. This method is "unsupervised" in the sense that it makes no direct reference to reward or the VF estimate. We introduce an algorithm based upon this idea which adapts a state aggregation approximation architecture on-line. A common method of scoring a VF estimate is to weight the squared Bellman error of each state-action by the probability of that state-action occurring. Adopting this scoring method, and assuming $S$ states, we demonstrate theoretically that - provided (1) the number of cells $X$ in the state aggregation architecture is of order $\sqrt{S}\log_2{S}\ln{S}$ or greater, (2) the policy and transition function are close to deterministic, and (3) the prior for the transition function is uniformly distributed - our algorithm, used in conjunction with a suitable RL algorithm, can guarantee a score which is arbitrarily close to zero as $S$ becomes large. It is able to do this despite having only $O(X \log_2S)$ space complexity and negligible time complexity. The results take advantage of certain properties of the stationary distributions of Markov chains.

📄 PDF Abstract BibTeX arXiv:1703.01026

Code (0)

등록된 구현이 없습니다.

Tasks

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

From Biological Synapses to Intelligent Robots

2022-02-25 · Birgitta Dresp-Langley

This review explores biologically inspired learning as a model for intelligent robot control and sensing technology on the basis of specific examples. Hebbian synaptic learning is discussed as a functionally relevant mod…

Disentangled Predictive Representation for Meta-Reinforcement Learning

2021-06-13 · ICML Workshop URL 2021 7 · Sephora Madjiheurem, Laura Toni

A major challenge in reinforcement learning is the design of agents that are able to generalize across tasks that share common dynamics. A viable solution is meta-reinforcement learning, which identifies common struc…

Meta Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

From Feature Learning to Spectral Basis Learning: A Unifying and Flexible Framework for Efficient and Robust Shape Matching

2026-03-24 · Feifan Luo, Hongyang Chen arxiv

Shape matching is a fundamental task in computer graphics and vision, with deep functional maps becoming a prominent paradigm. However, existing methods primarily focus on learning informative feature representations by …

Neural Approximation and Its Applications

2026-03-04 · Wei-Hao Wu, Ting-Zhu Huang, Xi-Le Zhao, Yisi Luo 외 arxiv

Multivariate function approximation is a fundamental problem in machine learning. Classic multivariate function approximations rely on hand-crafted basis functions (e.g., polynomial basis and Fourier basis), which limits…

Unsupervised Reinforcement Adaptation for Class-Imbalanced TextClassification

2022-01-16 · ACL ARR January 2022 1 · Anonymous

Unsupervised domain adaptation (UDA) augment model performance with only accessible annotations from the source domain and unlabeled data from the target domain. Existing state-of-the-art UDA models learn domain-invarian…

Domain Adaptation reinforcement-learning Reinforcement Learning (RL)text-classification+2