More Benefits of Being Distributional: Second-Order Bounds for Reinforcement Learning

2024-02-11 · Kaiwen Wang, Owen Oertell, Alekh Agarwal, Nathan Kallus, Wen Sun

In this paper, we prove that Distributional Reinforcement Learning (DistRL), which learns the return distribution, can obtain second-order bounds in both online and offline RL in general settings with function approximation. Second-order bounds are instance-dependent bounds that scale with the variance of return, which we prove are tighter than the previously known small-loss bounds of distributional RL. To the best of our knowledge, our results are the first second-order bounds for low-rank MDPs and for offline RL. When specializing to contextual bandits (one-step RL problem), we show that a distributional learning based optimism algorithm achieves a second-order worst-case regret bound, and a second-order gap dependent bound, simultaneously. We also empirically demonstrate the benefit of DistRL in contextual bandits on real-world datasets. We highlight that our analysis with DistRL is relatively simple, follows the general framework of optimism in the face of uncertainty and does not require weighted regression. Our results suggest that DistRL is a promising framework for obtaining second-order bounds in general RL settings, thus further reinforcing the benefits of DistRL.

📄 PDF Abstract BibTeX arXiv:2402.07198

Code (0)

등록된 구현이 없습니다.

Tasks

Distributional Reinforcement LearningMulti-Armed Bandits Offline RL

The Central Role of the Loss Function in Reinforcement Learning

2024-09-19 · Kaiwen Wang, Nathan Kallus, Wen Sun

This paper illustrates the central role of loss functions in data-driven decision making, providing a comprehensive survey on their influence in cost-sensitive classification (CSC) and reinforcement learning (RL). We dem…

Decision Making reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

The Benefits of Being Categorical Distributional: Uncertainty-aware Regularized Exploration in Reinforcement Learning

2021-10-07 · Ke Sun, Yingnan Zhao, Enze Shi, Yafei Wang 외

The remarkable empirical performance of distributional reinforcement learning (RL) has garnered increasing attention to understanding its theoretical advantages over classical RL. By decomposing the categorical distribut…

Atari Games Attribute Distributional Reinforcement Learning reinforcement-learning+1

The Benefits of Being Distributional: Small-Loss Bounds for Reinforcement Learning

2023-05-25 · NeurIPS 2023 11 · Kaiwen Wang, Kevin Zhou, Runzhe Wu, Nathan Kallus 외

While distributional reinforcement learning (DistRL) has been empirically effective, the question of when and why it is better than vanilla, non-distributional RL has remained unanswered. This paper explains the benefits…

Distributional Reinforcement Learning Offline RL reinforcement-learning

4-bit Shampoo for Memory-Efficient Network Training

2024-05-28 · Sike Wang, Pan Zhou, Jia Li, Hua Huang

Second-order optimizers, maintaining a matrix termed a preconditioner, are superior to first-order optimizers in both theory and practice. The states forming the preconditioner and its inverse root restrict the maximum s…

image-classification Image Classification Language Modeling Language Modelling+1

Robust Adaptive Beamforming Maximizing the Worst-Case SINR over Distributional Uncertainty Sets for Random INC Matrix and Signal Steering Vector

2021-10-16 · Yongwei Huang, Wenzheng Yang, Sergiy A. Vorobyov

The robust adaptive beamforming (RAB) problem is considered via the worst-case signal-to-interference-plus-noise ratio (SINR) maximization over distributional uncertainty sets for the random interference-plus-noise covar…