D-SPEAR: Dual-Stream Prioritized Experience Adaptive Replay for Stable Reinforcement Learning in Robotic Manipulation

2026-03-28 · Yu Zhang, Karl Mason arxiv

Robotic manipulation remains challenging for reinforcement learning due to contact-rich dynamics, long horizons, and training instability. Although off-policy actor-critic algorithms such as SAC and TD3 perform well in simulation, they often suffer from policy oscillations and performance collapse in realistic settings, partly due to experience replay strategies that ignore the differing data requirements of the actor and the critic. We propose D-SPEAR: Dual-Stream Prioritized Experience Adaptive Replay, a replay framework that decouples actor and critic sampling while maintaining a shared replay buffer. The critic leverages prioritized replay for efficient value learning, whereas the actor is updated using low-error transitions to stabilize policy optimization. An adaptive anchor mechanism balances uniform and prioritized sampling based on the coefficient of variation of TD errors, and a Huber-based critic objective further improves robustness under heterogeneous reward scales. We evaluate D-SPEAR on challenging robotic manipulation tasks from the robosuite benchmark, including Block-Lifting and Door-Opening. Results demonstrate that D-SPEAR consistently outperforms strong off-policy baselines, including SAC, TD3, and DDPG, in both final performance and training stability, with ablation studies confirming the complementary roles of the actorside and critic-side replay streams.

📄 PDF Abstract BibTeX arXiv:2603.27346

Code (0)

등록된 구현이 없습니다.

Tasks

Reinforcement Learning

Double Prioritized State Recycled Experience Replay

2020-07-08 · Fanchen Bu, Dong Eui Chang

Experience replay enables online reinforcement learning agents to store and reuse the previous experiences of interacting with the environment. In the original method, the experiences are sampled and replayed uniformly a…

Atari Games

Return Dispersion as an Estimator of Learning Potential for Prioritized Level Replay

2021-09-22 · NeurIPS Workshop ICBINB 2021 12 · Iryna Korshunova, Minqi Jiang, Jack Parker-Holder, Tim Rocktäschel 외

Prioritized Level Replay (PLR) has been shown to induce adaptive curricula that improve the sample-efficiency and generalization of reinforcement learning policies in environments featuring multiple tasks or levels. PLR …

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

A novel DDPG method with prioritized experience replay

2017-10-01 · IEEE International Conference on Systems, Man and Cybernetics (SMC) 2017 10 · Yuenan Hou, Lifeng Liu, Qing Wei, Xudong Xu 외

Recently, a state-of-the-art algorithm, called deep deterministic policy gradient (DDPG), has achieved good performance in many continuous control tasks in the MuJoCo simulator. To further improve the efficiency of the e…

continuous-control Continuous Control MuJoCo OpenAI Gym

Adaptive KDE for Real-Time Thresholding: Prioritized Queues for Financial Crime Investigation

2026-01-20 · Danny Butvinik, Nana Boateng, Achi Hackmon arxiv

We study the problem of converting a continuous stream of risk scores into stable decision thresholds under non-stationary score distributions. This problem arises in a wide range of detection systems where scores must b…

Prioritized Experience Replay

2015-11-18 · Tom Schaul, John Quan, Ioannis Antonoglou, David Silver

Experience replay lets online reinforcement learning agents remember and reuse experiences from the past. In prior work, experience transitions were uniformly sampled from a replay memory. However, this approach simply r…

Atari Games reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)