The Central Role of the Loss Function in Reinforcement Learning

2024-09-19 · Kaiwen Wang, Nathan Kallus, Wen Sun

This paper illustrates the central role of loss functions in data-driven decision making, providing a comprehensive survey on their influence in cost-sensitive classification (CSC) and reinforcement learning (RL). We demonstrate how different regression loss functions affect the sample efficiency and adaptivity of value-based decision making algorithms. Across multiple settings, we prove that algorithms using the binary cross-entropy loss achieve first-order bounds scaling with the optimal policy's cost and are much more efficient than the commonly used squared loss. Moreover, we prove that distributional algorithms using the maximum likelihood loss achieve second-order bounds scaling with the policy variance and are even sharper than first-order bounds. This in particular proves the benefits of distributional RL. We hope that this paper serves as a guide analyzing decision making algorithms with varying loss functions, and can inspire the reader to seek out better loss functions to improve any decision making algorithm.

📄 PDF Abstract BibTeX arXiv:2409.12799

Code (0)

등록된 구현이 없습니다.

Tasks

Decision Making reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

A Kernel Loss for Solving the Bellman Equation

2019-05-25 · NeurIPS 2019 12 · Yihao Feng, Lihong Li, Qiang Liu

Value function learning plays a central role in many state-of-the-art reinforcement-learning algorithms. Many popular algorithms like Q-learning do not optimize any objective function, but are fixed-point iterations of s…

Q-Learning Reinforcement Learning Reinforcement Learning (RL)

Upper Confidence Primal-Dual Reinforcement Learning for CMDP with Adversarial Loss

2020-03-02 · NeurIPS 2020 12 · Shuang Qiu, Xiaohan Wei, Zhuoran Yang, Jieping Ye 외

We consider online learning for episodic stochastically constrained Markov decision processes (CMDPs), which plays a central role in ensuring the safety of reinforcement learning. Here the loss function can vary arbitrar…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Emergence of Roles in Robotic Teams with Model Sharing and Limited Communication

2025-05-01 · Ian O'Flynn, Harun Šiljak

We present a reinforcement learning strategy for use in multi-agent foraging systems in which the learning is centralised to a single agent and its model is periodically disseminated among the population of non-learning …

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning

Gaussian-Mixture-Model Q-Functions for Reinforcement Learning by Riemannian Optimization

2024-09-06 · Minh Vu, Konstantinos Slavakis

This paper establishes a novel role for Gaussian-mixture models (GMMs) as functional approximators of Q-function losses in reinforcement learning (RL). Unlike the existing RL literature, where GMMs play their typical rol…

Reinforcement Learning (RL)Riemannian optimization

Enhancing Multi-Agent Collaboration with Attention-Based Actor-Critic Policies

2025-07-30 · Hugo Garrido-Lestache Belinchon, Jeremy Kedziora arxiv

This paper introduces Team-Attention-Actor-Critic (TAAC), a reinforcement learning algorithm designed to enhance multi-agent collaboration in cooperative environments. TAAC employs a Centralized Training/Centralized Exec…

Reinforcement Learning