Deterministic Implementations for Reproducibility in Deep Reinforcement Learning

2018-09-15 · Prabhat Nagarajan, Garrett Warnell, Peter Stone

While deep reinforcement learning (DRL) has led to numerous successes in recent years, reproducing these successes can be extremely challenging. One reproducibility challenge particularly relevant to DRL is nondeterminism in the training process, which can substantially affect the results. Motivated by this challenge, we study the positive impacts of deterministic implementations in eliminating nondeterminism in training. To do so, we consider the particular case of the deep Q-learning algorithm, for which we produce a deterministic implementation by identifying and controlling all sources of nondeterminism in the training process. One by one, we then allow individual sources of nondeterminism to affect our otherwise deterministic implementation, and measure the impact of each source on the variance in performance. We find that individual sources of nondeterminism can substantially impact the performance of agent, illustrating the benefits of deterministic implementations. In addition, we also discuss the important role of deterministic implementations in achieving exact replicability of results.

📄 PDF Abstract BibTeX arXiv:1809.05676

Code (1)

prabhatnagarajan/repro_dqn 공식 구현 pytorch

Tasks

Deep Reinforcement LearningQ-Learningreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

BenchMARL: Benchmarking Multi-Agent Reinforcement Learning

2023-12-03 · Matteo Bettini, Amanda Prorok, Vincent Moens

The field of Multi-Agent Reinforcement Learning (MARL) is currently facing a reproducibility crisis. While solutions for standardized reporting have been proposed to address the issue, we still lack a benchmarking tool t…

Benchmarking Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning+1

The Effect of Third Party Implementations on Reproducibility

2023-07-27 · Balázs Hidasi, Ádám Tibor Czapp

Reproducibility of recommender systems research has come under scrutiny during recent years. Along with works focusing on repeating experiments with certain algorithms, the research community has also started discussing …

Recommendation Systems

AdversariaLLM: A Unified and Modular Toolbox for LLM Robustness Research

2025-11-06 · Tim Beyer, Jonas Dornbusch, Jakob Steimle, Moritz Ladenburger 외 arxiv

The rapid expansion of research on Large Language Model (LLM) safety and robustness has produced a fragmented and oftentimes buggy ecosystem of implementations, datasets, and evaluation methods. This fragmentation makes …

Adversarial Attack

Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control

2017-08-10 · Riashat Islam, Peter Henderson, Maziar Gomrokchi, Doina Precup

Policy gradient methods in reinforcement learning have become increasingly prevalent for state-of-the-art performance in continuous control tasks. Novel methods typically benchmark against a few key algorithms such as de…

continuous-control Continuous Control Deep Reinforcement Learning Policy Gradient Methods+3

DASH: Deterministic Attention Scheduling for High-throughput Reproducible LLM Training

2026-01-29 · Xinwei Qiang, Hongmin Chen, Shixuan Sun, Jingwen Leng 외 arxiv

Determinism is indispensable for reproducibility in large language model (LLM) training, yet it often exacts a steep performance cost. In widely used attention implementations such as FlashAttention-3, the deterministic …