Provable Risk-Sensitive Distributional Reinforcement Learning with General Function Approximation
In the realm of reinforcement learning (RL), accounting for risk is crucial for making decisions under uncertainty, particularly in applications where safety and reliability are paramount. In this paper, we introduce a general framework on Risk-Sensitive Distributional Reinforcement Learning (RS-DisRL), with static Lipschitz Risk Measures (LRM) and general function approximation. Our framework covers a broad class of risk-sensitive RL, and facilitates analysis of the impact of estimation functions on the effectiveness of RSRL strategies and evaluation of their sample complexity. We design two innovative meta-algorithms: \texttt{RS-DisRL-M}, a model-based strategy for model-based function approximation, and \texttt{RS-DisRL-V}, a model-free approach for general value function approximation. With our novel estimation techniques via Least Squares Regression (LSR) and Maximum Likelihood Estimation (MLE) in distributional RL with augmented Markov Decision Process (MDP), we derive the first $\widetilde{\mathcal{O}}(\sqrt{K})$ dependency of the regret upper bound for RSRL with static LRM, marking a pioneering contribution towards statistically efficient algorithms in this domain.
Code (0)
등록된 구현이 없습니다.
Tasks
Distributional Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Is Risk-Sensitive Reinforcement Learning Properly Resolved?
Due to the nature of risk management in learning applicable policies, risk-sensitive reinforcement learning (RSRL) has been realized as an important direction. RSRL is usually achieved by learning risk-sensitive objectiv…
Distributional Reinforcement LearningManagementQ-Learningreinforcement-learning+1Policy Gradient Methods for Risk-Sensitive Distributional Reinforcement Learning with Provable Convergence
Risk-sensitive reinforcement learning (RL) is crucial for maintaining reliable performance in high-stakes applications. While traditional RL methods aim to learn a point estimate of the random cumulative cost, distributi…
Distributional Reinforcement LearningPolicy Gradient MethodsReinforcement Learning (RL)Bridging Distributional and Risk-sensitive Reinforcement Learning with Provable Regret Bounds
We study the regret guarantee for risk-sensitive reinforcement learning (RSRL) via distributional reinforcement learning (DRL) methods. In particular, we consider finite episodic Markov decision processes whose objective…
Computational EfficiencyDistributional Reinforcement Learningreinforcement-learningReinforcement Learning+1Distributional Model Equivalence for Risk-Sensitive Reinforcement Learning
We consider the problem of learning models for risk-sensitive reinforcement learning. We theoretically demonstrate that proper value equivalence, a method of learning models which can be used to plan optimally in the ris…
Distributional Reinforcement Learningmodelreinforcement-learningReinforcement LearningRisk-Sensitive Policy with Distributional Reinforcement Learning
Classical reinforcement learning (RL) techniques are generally concerned with the design of decision-making policies driven by the maximisation of the expected outcome. Nevertheless, this approach does not take into cons…
Decision MakingDistributional Reinforcement Learningreinforcement-learningReinforcement Learning+2