An Optimal Policy for Learning Controllable Dynamics by Exploration
Controllable Markov chains describe the dynamics of sequential decision making tasks and are the central component in optimal control and reinforcement learning. In this work, we give the general form of an optimal policy for learning controllable dynamics in an unknown environment by exploring over a limited time horizon. This policy is simple to implement and efficient to compute, and allows an agent to ``learn by exploring" as it maximizes its information gain in a greedy fashion by selecting controls from a constraint set that changes over time during exploration. We give a simple parameterization for the set of controls, and present an algorithm for finding an optimal policy. The reason for this policy is due to the existence of certain types of states that restrict control of the dynamics; such as transient states, absorbing states, and non-backtracking states. We show why the occurrence of these states makes a non-stationary policy essential for achieving optimal exploration. Six interesting examples of controllable dynamics are treated in detail. Policy optimality is demonstrated using counting arguments, comparing with suboptimal policies, and by making use of a sequential improvement property from dynamic programming.
Code (0)
등록된 구현이 없습니다.
Tasks
Reinforcement LearningDecision MakingSimilar Papers 제목 키워드 기반
Learning controllable dynamics through informative exploration
Environments with controllable dynamics are usually understood in terms of explicit models. However, such models are not always available, but may sometimes be learned by exploring an environment. In this work, we invest…
Reinforcement LearningControllable Exploration in Hybrid-Policy RLVR for Multi-Modal Reasoning
Reinforcement Learning with verifiable rewards (RLVR) has emerged as a primary learning paradigm for enhancing the reasoning capabilities of multi-modal large language models (MLLMs). However, during RL training, the eno…
Reinforcement LearningGlobal optimality of softmax policy gradient with single hidden layer neural networks in the mean-field regime
We study the problem of policy optimization for infinite-horizon discounted Markov Decision Processes with softmax policy and nonlinear function approximation trained with policy gradient algorithms. We concentrate on th…
Model Generation with Provable Coverability for Offline Reinforcement Learning
Model-based offline optimization with dynamics-aware policy provides a new perspective for policy learning and out-of-distribution generalization, where the learned policy could adapt to different dynamics enumerated at …
Offline RLOut-of-Distribution Generalizationreinforcement-learningReinforcement Learning+1Distribution-Centric Policy Optimization Dominates Exploration-Exploitation Trade-off
The exploration-exploitation (EE) trade-off is a central challenge in reinforcement learning (RL) for large language models (LLMs). With Group Relative Policy Optimization (GRPO), training tends to be exploitation driven…
Reinforcement Learning