Risk-Sensitive Stochastic Optimal Control as Rao-Blackwellized Markovian Score Climbing
Stochastic optimal control of dynamical systems is a crucial challenge in sequential decision-making. Recently, control-as-inference approaches have had considerable success, providing a viable risk-sensitive framework to address the exploration-exploitation dilemma. Nonetheless, a majority of these techniques only invoke the inference-control duality to derive a modified risk objective that is then addressed within a reinforcement learning framework. This paper introduces a novel perspective by framing risk-sensitive stochastic control as Markovian score climbing under samples drawn from a conditional particle filter. Our approach, while purely inference-centric, provides asymptotically unbiased estimates for gradient-based policy optimization with optimal importance weighting and no explicit value function learning. To validate our methodology, we apply it to the task of learning neural non-Gaussian feedback policies, showcasing its efficacy on numerical benchmarks of stochastic dynamical systems.
Code (1)
Tasks
Decision MakingSequential Decision MakingSimilar Papers 제목 키워드 기반
Risk-Sensitive Q-Learning in Continuous Time with Application to Dynamic Portfolio Selection
This paper studies the problem of risk-sensitive reinforcement learning (RSRL) in continuous time, where the environment is characterized by a controllable stochastic differential equation (SDE) and the objective is a po…
Reinforcement LearningA stochastic control perspective on term structure models with roll-over risk
In this paper, we consider a generic interest rate market in the presence of roll-over risk, which generates spreads in spot/forward term rates. We do not require classical absence of arbitrage and rely instead on a mini…
Risk-sensitive control as inference with Rényi divergence
This paper introduces the risk-sensitive control as inference (RCaI) that extends CaI by using R\'{e}nyi divergence variational inference. RCaI is shown to be equivalent to log-probability regularized risk-sensitive cont…
Reinforcement Learning (RL)Variational InferenceDistributionally Robust Path Integral Control
We consider a continuous-time continuous-space stochastic optimal control problem, where the controller lacks exact knowledge of the underlying diffusion process, relying instead on a finite set of historical disturbance…
On the Global Convergence of Risk-Averse Policy Gradient Methods with Expected Conditional Risk Measures
Risk-sensitive reinforcement learning (RL) has become a popular tool for controlling the risk of uncertain outcomes and ensuring reliable performance in highly stochastic sequential decision-making problems. While Policy…
Decision MakingPolicy Gradient MethodsReinforcement Learning (RL)Sequential Decision Making