How to Discount Deep Reinforcement Learning: Towards New Dynamic Strategies

2015-12-07 · Vincent François-Lavet, Raphael Fonteneau, Damien Ernst

Using deep neural nets as function approximator for reinforcement learning tasks have recently been shown to be very powerful for solving problems approaching real-world complexity. Using these results as a benchmark, we discuss the role that the discount factor may play in the quality of the learning process of a deep Q-network (DQN). When the discount factor progressively increases up to its final value, we empirically show that it is possible to significantly reduce the number of learning steps. When used in conjunction with a varying learning rate, we empirically show that it outperforms original DQN on several experiments. We relate this phenomenon with the instabilities of neural networks when they are used in an approximate Dynamic Programming setting. We also describe the possibility to fall within a local optimum during the learning process, thus connecting our discussion with the exploration/exploitation dilemma.

📄 PDF Abstract BibTeX arXiv:1512.02011

Code (0)

등록된 구현이 없습니다.

Tasks

Deep Reinforcement Learningreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Convolution A convolution is a type of matrix operation, consisting of a kernel, a small matrix of weights, that slides over input data performing element-wise multiplication with the…

DQN A DQN, or Deep Q-Network, approximates a state-value function in a Q-Learning framework with a neural network. In the Atari…

Discounted Pseudocosts in MILP

2024-07-07 · Krunal Kishor Patel

In this article, we introduce the concept of discounted pseudocosts, inspired by discounted total reward in reinforcement learning, and explore their application in mixed-integer linear programming (MILP). Traditional ps…

reinforcement-learning Reinforcement Learning

Examining average and discounted reward optimality criteria in reinforcement learning

2021-07-03 · Vektor Dewanto, Marcus Gallagher

In reinforcement learning (RL), the goal is to obtain an optimal policy, for which the optimality criterion is fundamentally important. Two major optimality criteria are average and discounted rewards. While the latter i…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Market Making Strategies with Reinforcement Learning

2025-07-24 · Óscar Fernández Vicente arxiv

This thesis presents the results of a comprehensive research project focused on applying Reinforcement Learning (RL) to the problem of market making in financial markets. Market makers (MMs) play a fundamental role in pr…

Reinforcement Learning

Loss Dynamics of Temporal Difference Reinforcement Learning

2023-07-10 · NeurIPS 2023 11 · Blake Bordelon, Paul Masset, Henry Kuo, Cengiz Pehlevan

Reinforcement learning has been successful across several applications in which agents have to learn to act in environments with sparse feedback. However, despite this empirical success there is still a lack of theoretic…