Inverse Policy Evaluation for Value-based Sequential Decision-making
Value-based methods for reinforcement learning lack generally applicable ways to derive behavior from a value function. Many approaches involve approximate value iteration (e.g., $Q$-learning), and acting greedily with respect to the estimates with an arbitrary degree of entropy to ensure that the state-space is sufficiently explored. Behavior based on explicit greedification assumes that the values reflect those of \textit{some} policy, over which the greedy policy will be an improvement. However, value-iteration can produce value functions that do not correspond to \textit{any} policy. This is especially relevant in the function-approximation regime, when the true value function can't be perfectly represented. In this work, we explore the use of \textit{inverse policy evaluation}, the process of solving for a likely policy given a value function, for deriving behavior from a value function. We provide theoretical and empirical results to show that inverse policy evaluation, combined with an approximate value iteration algorithm, is a feasible method for value-based control.
Code (0)
등록된 구현이 없습니다.
Tasks
Decision MakingQ-LearningSequential Decision MakingSimilar Papers 제목 키워드 기반
Statistical Inference of the Value Function for Reinforcement Learning in Infinite Horizon Settings
Reinforcement learning is a general technique that allows an agent to learn an optimal policy and interact with an environment in sequential decision making problems. The goodness of a policy is measured by its value fun…
Decision Makingreinforcement-learningReinforcement LearningReinforcement Learning (RL)+1Sequential Bayesian Optimal Experimental Design in Infinite Dimensions via Policy Gradient Reinforcement Learning
Sequential Bayesian optimal experimental design (SBOED) for PDE-governed inverse problems is computationally challenging, especially for infinite-dimensional random field parameters. High-fidelity approaches require repe…
Reinforcement LearningInsurance Pricing Optimization via Off-Policy Evaluation
Traditional insurance pricing relies on risk-based principles that ensure actuarial fairness and solvency but do not explicitly account for policyholders' price sensitivity. We formulate insurance pricing as a decision-m…
Interactively Teaching an Inverse Reinforcement Learner with Limited Feedback
We study the problem of teaching via demonstrations in sequential decision-making tasks. In particular, we focus on the situation when the teacher has no access to the learner's model and policy, and the feedback from th…
Active LearningDecision MakingSequential Decision MakingIPD: Boosting Sequential Policy with Imaginary Planning Distillation in Offline Reinforcement Learning
Decision transformer based sequential policies have emerged as a powerful paradigm in offline reinforcement learning (RL), yet their efficacy remains constrained by the quality of static datasets and inherent architectur…
Reinforcement LearningOffline RL