Reinforcement Learning with Quasi-Hyperbolic Discounting

2024-09-16 · S. R. Eshwar, Mayank Motwani, Nibedita Roy, Gugan Thoppe

Reinforcement learning has traditionally been studied with exponential discounting or the average reward setup, mainly due to their mathematical tractability. However, such frameworks fall short of accurately capturing human behavior, which has a bias towards immediate gratification. Quasi-Hyperbolic (QH) discounting is a simple alternative for modeling this bias. Unlike in traditional discounting, though, the optimal QH-policy, starting from some time $t_1,$ can be different to the one starting from $t_2.$ Hence, the future self of an agent, if it is naive or impatient, can deviate from the policy that is optimal at the start, leading to sub-optimal overall returns. To prevent this behavior, an alternative is to work with a policy anchored in a Markov Perfect Equilibrium (MPE). In this work, we propose the first model-free algorithm for finding an MPE. Using a two-timescale analysis, we show that, if our algorithm converges, then the limit must be an MPE. We also validate this claim numerically for the standard inventory system with stochastic demands. Our work significantly advances the practical application of reinforcement learning.

📄 PDF Abstract BibTeX arXiv:2409.10583

Code (0)

등록된 구현이 없습니다.

Tasks

reinforcement-learning Reinforcement Learning

A simple framework for the axiomatization of exponential and quasi-hyperbolic discounting

2015-11-19

The main goal of this paper is to investigate which normative requirements, or axioms, lead to exponential and quasi-hyperbolic forms of discounting. Exponential discounting has a well-established axiomatic foundation or…

Teaching Precommitted Agents: Model-Free Policy Evaluation and Control in Quasi-Hyperbolic Discounted MDPs

2025-09-07 · S. R. Eshwar arxiv

Time-inconsistent preferences, where agents favor smaller-sooner over larger-later rewards, are a key feature of human and animal decision-making. Quasi-Hyperbolic (QH) discounting provides a simple yet powerful model fo…

Reinforcement Learning

Hyperbolically-Discounted Reinforcement Learning on Reward-Punishment Framework

2021-06-03 · Taisuke Kobayashi

This paper proposes a new reinforcement learning with hyperbolic discounting. Combining a new temporal difference error with the hyperbolic discounting in recursive manner and reward-punishment framework, a new scheme to…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Hyperbolic Discounting and Learning over Multiple Horizons

2019-02-19 · ICLR 2020 1 · William Fedus, Carles Gelada, Yoshua Bengio, Marc G. Bellemare 외

Reinforcement learning (RL) typically defines a discount factor as part of the Markov Decision Process. The discount factor values future rewards by an exponential scheme that leads to theoretical convergence guarantees …

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Revealed preferences for dynamically inconsistent models

2023-05-23 · Federico Echenique, Gerelt Tserenjigmid

We study the testable implications of models of dynamically inconsistent choices when planned choices are unobservable, and thus only "on path" data is available. First, we discuss the approach in Blow, Browning and Craw…