Actor-Critic Algorithm for Dynamic Expectile and CVaR

2026-05-08 · Yudong Luo, Erick Delage arxiv

Optimizing dynamic risk with stochastic policies is challenging in both policy updates and value learning. The former typically requires transition perturbation, while the latter may rely on model-based approaches. To address these challenges, we propose a surrogate policy gradient without transition perturbation under softmax policy parameterization. We further develop model-free value learning methods for dynamic expectile and conditional value-at-risk by leveraging elicitability. Finally, inspired by Expected SARSA and Expected Policy Gradient, a model-free off-policy actor-critic algorithm is constructed. Empirical results in domains with verifiable risk-averse behavior show that our algorithm can learn risk-averse policy and consistently outperforms other existing methods.

📄 PDF Abstract BibTeX arXiv:2605.07857

Code (0)

등록된 구현이 없습니다.

Expectile Quadrangle and Applications

2023-06-28 · Viktor Kuzmenko, Anton Malandii, Stan Uryasev

The paper explores the concept of the \emph{expectile risk measure} within the framework of the Fundamental Risk Quadrangle (FRQ) theory. According to the FRQ theory, a quadrangle comprises four stochastic functions asso…

Management

Safe Langevin Soft Actor Critic

2026-01-31 · Mahesh Keswani, Samyak Jain, Raunak P. Bhattacharyya arxiv

Balancing reward and safety in constrained reinforcement learning remains challenging due to poor generalization from sharp value minima and inadequate handling of heavy-tailed risk distribution. We introduce Safe Langev…

Reinforcement Learning

Moderate Actor-Critic Methods: Controlling Overestimation Bias via Expectile Loss

2025-04-14 · Ukjo Hwang, SongNam Hong

Overestimation is a fundamental characteristic of model-free reinforcement learning (MF-RL), arising from the principles of temporal difference learning and the approximation of the Q-function. To address this challenge,…

Bootstrapping Expectiles in Reinforcement Learning

2024-06-06 · Pierre Clavier, Emmanuel Rachelson, Erwan Le Pennec, Matthieu Geist

Many classic Reinforcement Learning (RL) algorithms rely on a Bellman operator, which involves an expectation over the next states, leading to the concept of bootstrapping. To introduce a form of pessimism, we propose to…

Q-Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Expectile Matrix Factorization for Skewed Data Analysis

2016-06-07 · Rui Zhu, Di Niu, Linglong Kong, Zongpeng Li

Matrix factorization is a popular approach to solving matrix estimation problems based on partial observations. Existing matrix factorization is based on least squares and aims to yield a low-rank matrix to interpret the…