Position: Benchmarking is Limited in Reinforcement Learning Research

2024-06-23 · Scott M. Jordan, Adam White, Bruno Castro da Silva, Martha White, Philip S. Thomas

Novel reinforcement learning algorithms, or improvements on existing ones, are commonly justified by evaluating their performance on benchmark environments and are compared to an ever-changing set of standard algorithms. However, despite numerous calls for improvements, experimental practices continue to produce misleading or unsupported claims. One reason for the ongoing substandard practices is that conducting rigorous benchmarking experiments requires substantial computational time. This work investigates the sources of increased computation costs in rigorous experiment designs. We show that conducting rigorous performance benchmarks will likely have computational costs that are often prohibitive. As a result, we argue for using an additional experimentation paradigm to overcome the limitations of benchmarking.

📄 PDF Abstract BibTeX arXiv:2406.16241

Code (0)

등록된 구현이 없습니다.

Tasks

BenchmarkingPositionreinforcement-learning Reinforcement Learning

Methods 이 논문이 사용한 방법론

SET Dynamic Sparse Training method where weight mask is updated randomly periodically

Reinforcement Learning for Elliptical Cylinder Motion Control Tasks

2026-03-13 · Pawel Marczewski, Paulina Superczynska, Jakub Bernat, Szymon Szczesny arxiv

The control of devices with limited input always bring attention to solve by research due to its difficulty and non-trival solution. For instance, the inverted pendulum is benchmarking problem in control theory and machi…

Reinforcement Learning

Benchmarking Quantum Reinforcement Learning

2025-01-27 · Nico Meyer, Christian Ufrecht, George Yammine, Georgios Kontes 외

Benchmarking and establishing proper statistical validation metrics for reinforcement learning (RL) remain ongoing challenges, where no consensus has been established yet. The emergence of quantum computing and its poten…

Benchmarking reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1

A survey of benchmarking frameworks for reinforcement learning

2020-11-27 · Belinda Stapelberg, Katherine M. Malan

Reinforcement learning has recently experienced increased prominence in the machine learning community. There are many approaches to solving reinforcement learning problems with new techniques developed constantly. When …

Benchmarking reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1

Robotic Manipulation Datasets for Offline Compositional Reinforcement Learning

2023-07-13 · Marcel Hussing, Jorge A. Mendez, Anisha Singrodia, Cassandra Kent 외

Offline reinforcement learning (RL) is a promising direction that allows RL agents to pre-train on large datasets, avoiding the recurrence of expensive data collection. To advance the field, it is crucial to generate lar…

Benchmarking Offline RL reinforcement-learning Reinforcement Learning+1

RoAd-RL: A Unified Library and Benchmark for Robust Adversarial Reinforcement Learning

2026-06-29 · Adithya Mohan, Daniel Kriegl, Torsten Schön arxiv

Deep Reinforcement Learning (DRL) has achieved significant success in robotics and autonomous systems, yet remains vulnerable to adversarial perturbations that can severely degrade performance. Research in adversarial re…