Risk-sensitive Reinforcement Learning

2013-11-08 · Yun Shen, Michael J. Tobia, Tobias Sommer, Klaus Obermayer

We derive a family of risk-sensitive reinforcement learning methods for agents, who face sequential decision-making tasks in uncertain environments. By applying a utility function to the temporal difference (TD) error, nonlinear transformations are effectively applied not only to the received rewards but also to the true transition probabilities of the underlying Markov decision process. When appropriate utility functions are chosen, the agents' behaviors express key features of human behavior as predicted by prospect theory (Kahneman and Tversky, 1979), for example different risk-preferences for gains and losses as well as the shape of subjective probability curves. We derive a risk-sensitive Q-learning algorithm, which is necessary for modeling human behavior when transition probabilities are unknown, and prove its convergence. As a proof of principle for the applicability of the new framework we apply it to quantify human behavior in a sequential investment task. We find, that the risk-sensitive variant provides a significantly better fit to the behavioral data and that it leads to an interpretation of the subject's responses which is indeed consistent with prospect theory. The analysis of simultaneously measured fMRI signals show a significant correlation of the risk-sensitive TD error with BOLD signal change in the ventral striatum. In addition we find a significant correlation of the risk-sensitive Q-values with neural activity in the striatum, cingulate cortex and insula, which is not present if standard Q-values are used.

📄 PDF Abstract BibTeX arXiv:1311.2097

Code (0)

등록된 구현이 없습니다.

Tasks

Decision MakingQ-Learningreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Sequential Decision Making

Methods 이 논문이 사용한 방법론

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

Distributional Model Equivalence for Risk-Sensitive Reinforcement Learning

2023-07-04 · NeurIPS 2023 11 · Tyler Kastner, Murat A. Erdogdu, Amir-Massoud Farahmand

We consider the problem of learning models for risk-sensitive reinforcement learning. We theoretically demonstrate that proper value equivalence, a method of learning models which can be used to plan optimally in the ris…

Distributional Reinforcement Learning model reinforcement-learning Reinforcement Learning

Regret Bounds for Risk-Sensitive Reinforcement Learning

2022-10-11 · O. Bastani, Y. J. Ma, E. Shen, W. Xu

In safety-critical applications of reinforcement learning such as healthcare and robotics, it is often desirable to optimize risk-sensitive objectives that account for tail outcomes rather than expected reward. We prove …

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Is Risk-Sensitive Reinforcement Learning Properly Resolved?

2023-07-02 · Ruiwen Zhou, Minghuan Liu, Kan Ren, Xufang Luo 외

Due to the nature of risk management in learning applicable policies, risk-sensitive reinforcement learning (RSRL) has been realized as an important direction. RSRL is usually achieved by learning risk-sensitive objectiv…

Distributional Reinforcement Learning Management Q-Learning reinforcement-learning+1