High-Throughput Distributed Reinforcement Learning via Adaptive Policy Synchronization

2025-07-15 · Rodney Lafuente-Mercado

Scaling reinforcement learning (RL) workloads often requires distributing environment simulation across compute clusters. Existing frameworks entangle simulation, learning logic, and orchestration into monolithic systems, limiting modularity and reusability. We present ClusterEnv, a lightweight, learner-agnostic interface for distributed environment execution that mirrors the Gymnasium API. ClusterEnv introduces the DETACH pattern, which decouples simulation from training by offloading reset() and step() operations to remote workers while keeping learning centralized. To address policy staleness in distributed execution, we propose Adaptive Actor Policy Synchronization (AAPS), a divergence-triggered update mechanism that reduces synchronization overhead without sacrificing performance. ClusterEnv integrates cleanly into existing RL pipelines, supports both on-policy and off-policy methods, and requires minimal code changes. Experiments on discrete control tasks demonstrate that AAPS achieves high sample efficiency with significantly fewer weight updates. Source code is available at https://github.com/rodlaf/ClusterEnv.

📄 PDF Abstract BibTeX arXiv:2507.10990

Code (1)

rodlaf/clusterenv 공식 구현 pytorch

Tasks

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Combining Contention-Based Spectrum Access and Adaptive Modulation using Deep Reinforcement Learning

2021-09-24 · Akash Doshi, Jeffrey G. Andrews

The use of unlicensed spectrum for cellular systems to mitigate spectrum scarcity has led to the development of intelligent adaptive approaches to spectrum access that improve upon traditional carrier sensing and listen-…

Deep Reinforcement Learning Fairness reinforcement-learning Reinforcement Learning (RL)

Reinforcement Learning Based Approaches to Adaptive Context Caching in Distributed Context Management Systems

2022-12-22 · Shakthi Weerasinghe, Arkady Zaslavsky, Seng W. Loke, Amin Abken 외

Performance metrics-driven context caching has a profound impact on throughput and response time in distributed context management systems for real-time context queries. This paper proposes a reinforcement learning based…

Management reinforcement-learning Reinforcement Learning (RL)

Distributed Proximal Policy Optimization for Contention-Based Spectrum Access

2021-10-07 · Akash Doshi, Jeffrey G. Andrews

The increasing number of wireless devices operating in unlicensed spectrum motivates the development of intelligent adaptive approaches to spectrum access that go beyond traditional carrier sensing. We develop a novel di…

Fairness

Sample Factory: Egocentric 3D Control from Pixels at 100000 FPS with Asynchronous Reinforcement Learning

2020-06-21 · ICML 2020 1 · Aleksei Petrenko, Zhehui Huang, Tushar Kumar, Gaurav Sukhatme 외

Increasing the scale of reinforcement learning experiments has allowed researchers to achieve unprecedented results in both training sophisticated agents for video games, and in sim-to-real transfer for robotics. Typical…

FPS Games General Reinforcement Learning GPU Multi-agent Reinforcement Learning+3

QoS and Jamming-Aware Wireless Networking Using Deep Reinforcement Learning

2019-10-13 · Nof Abuzainab, Tugba Erpek, Kemal Davaslioglu, Yalin E. Sagduyu 외

The problem of quality of service (QoS) and jamming-aware communications is considered in an adversarial wireless network subject to external eavesdropping and jamming attacks. To ensure robust communication against jamm…

Deep Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)