Reinforcement Learning via Conservative Agent for Environments with Random Delays
Real-world reinforcement learning applications are often hindered by delayed feedback from environments, which violates the Markov assumption and introduces significant challenges. Although numerous delay-compensating methods have been proposed for environments with constant delays, environments with random delays remain largely unexplored due to their inherent variability and unpredictability. In this study, we propose a simple yet robust agent for decision-making under random delays, termed the conservative agent, which reformulates the random-delay environment into its constant-delay equivalent. This transformation enables any state-of-the-art constant-delay method to be directly extended to the random-delay environments without modifying the algorithmic structure or sacrificing performance. We evaluate the conservative agent-based algorithm on continuous control tasks, and empirical results demonstrate that it significantly outperforms existing baseline algorithms in terms of asymptotic performance and sample efficiency.
Code (0)
등록된 구현이 없습니다.
Tasks
Reinforcement LearningContinuous ControlSimilar Papers 제목 키워드 기반
Adaptive Reinforcement Learning for Unobservable Random Delays
In standard Reinforcement Learning (RL) settings, the interaction between the agent and the environment is typically modeled as a Markov Decision Process (MDP), which assumes that the agent observes the system state inst…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Model-Based Reinforcement Learning under Random Observation Delays
Delays frequently occur in real-world environments, yet standard reinforcement learning (RL) algorithms often assume instantaneous perception of the environment. We study random sensor delays in POMDPs, where observation…
Reinforcement LearningReinforcement Learning with Random Delays
Action and observation delays commonly occur in many Reinforcement Learning applications, such as remote control scenarios. We study the anatomy of randomly delayed environments, and show that partially resampling trajec…
Anatomycontinuous-controlContinuous ControlMuJoCo+3Delay-Aware Multi-Agent Reinforcement Learning for Cooperative and Competitive Environments
Action and observation delays exist prevalently in the real-world cyber-physical systems which may pose challenges in reinforcement learning design. It is particularly an arduous task when handling multi-agent systems wh…
Autonomous VehiclesDeep Reinforcement LearningMulti-agent Reinforcement Learningreinforcement-learning+2Counterfactual Conservative Q Learning for Offline Multi-agent Reinforcement Learning
Offline multi-agent reinforcement learning is challenging due to the coupling effect of both distribution shift issue common in offline setting and the high dimension issue common in multi-agent setting, making the actio…
counterfactualMulti-agent Reinforcement LearningOffline RLQ-Learning+2