홈 › Papers

Balanced Q-learning: Combining the Influence of Optimistic and Pessimistic Targets

2021-11-03 · Thommen George Karimpanal, Hung Le, Majid Abdolshah, Santu Rana, Sunil Gupta, Truyen Tran, Svetha Venkatesh

The optimistic nature of the Q-learning target leads to an overestimation bias, which is an inherent problem associated with standard $Q-$learning. Such a bias fails to account for the possibility of low returns, particularly in risky scenarios. However, the existence of biases, whether overestimation or underestimation, need not necessarily be undesirable. In this paper, we analytically examine the utility of biased learning, and show that specific types of biases may be preferable, depending on the scenario. Based on this finding, we design a novel reinforcement learning algorithm, Balanced Q-learning, in which the target is modified to be a convex combination of a pessimistic and an optimistic term, whose associated weights are determined online, analytically. We prove the convergence of this algorithm in a tabular setting, and empirically demonstrate its superior learning performance in various environments.

📄 PDF Abstract BibTeX arXiv:2111.02787

Code (0)

등록된 구현이 없습니다.

Tasks

Q-Learning

Methods 이 논문이 사용한 방법론

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

Exploring Optimism and Pessimism in Twitter Using Deep Learning

2018-10-01 · EMNLP 2018 10 · Cornelia Caragea, Liviu P. Dinu, Bogdan Dumitru

Identifying optimistic and pessimistic viewpoints and users from Twitter is useful for providing better social support to those who need such support, and for minimizing the negative influence among users and maximizing …

Deep Learning

Efficient Reinforcement Learning via Decoupling Exploration and Utilization

2023-12-26 · Jingpu Yang, Helin Wang, Qirui Zhao, Zhecheng Shi 외

Reinforcement Learning (RL), recognized as an efficient learning approach, has achieved remarkable success across multiple fields and applications, including gaming, robotics, and autonomous vehicles. Classical single-ag…

Autonomous Vehicles MuJoCo reinforcement-learning Reinforcement Learning+1

Optimistic Exploration even with a Pessimistic Initialisation

2020-02-26 · ICLR 2020 1 · Tabish Rashid, Bei Peng, Wendelin Böhmer, Shimon Whiteson

Optimistic initialisation is an effective strategy for efficient exploration in reinforcement learning (RL). In the tabular case, all provably efficient model-free algorithms rely on it. However, model-free deep RL algor…

Efficient Exploration Q-Learning Reinforcement Learning Reinforcement Learning (RL)

Hyperparameter Tuning Through Pessimistic Bilevel Optimization

2024-12-04 · Meltem Apaydin Ustun, Liang Xu, Bo Zeng, Xiaoning Qian

Automated hyperparameter search in machine learning, especially for deep learning models, is typically formulated as a bilevel optimization problem, with hyperparameter values determined by the upper level and the model …

Bilevel Optimization Hyperparameter Optimization

POND: Pessimistic-Optimistic oNline Dispatching

2020-10-20 · Xin Liu, Bin Li, Pengyi Shi, Lei Ying

This paper considers constrained online dispatching with unknown arrival, reward and constraint distributions. We propose a novel online dispatching algorithm, named POND, standing for Pessimistic-Optimistic oNline Dispa…