Belief Projection-Based Reinforcement Learning for Environments with Delayed Feedback
We present a novel actor-critic algorithm for an environment with delayed feedback, which addresses the state-space explosion problem of conventional approaches. Conventional approaches use an augmented state constructed from the last observed state and actions executed since visiting the last observed state. Using the augmented state space, the correct Markov decision process for delayed environments can be constructed; however, this causes the state space to explode as the number of delayed timesteps increases, leading to slow convergence. Our proposed algorithm, called Belief-Projection-Based Q-learning (BPQL), addresses the state-space explosion problem by evaluating the values of the critic for which the input state size is equal to the original state-space size rather than that of the augmented one. We compare BPQL to traditional approaches in continuous control tasks and demonstrate that it significantly outperforms other algorithms in terms of asymptotic performance and sample efficiency. We also show that BPQL solves long-delayed environments, which conventional approaches are unable to do.
Code (1)
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Delayed homomorphic reinforcement learning for environments with delayed feedback
Reinforcement learning in real-world systems often involves delayed feedback, which breaks the Markov assumption and impedes both learning and control. Canonical augmentation-based approaches cause state-space explosion,…
Reinforcement LearningRewarding Beliefs, Not Actions: Consistency-Guided Credit Assignment for Long-Horizon Agents
Reinforcement learning from verifiable rewards (RLVR) is a promising paradigm for improving large language model (LLM) agents on long-horizon interactive tasks. However, in partially observable environments, incomplete o…
Reinforcement LearningStepScorer: Accelerating Reinforcement Learning with Step-wise Scoring and Psychological Regret Modeling
Reinforcement learning algorithms often suffer from slow convergence due to sparse reward signals, particularly in complex environments where feedback is delayed or infrequent. This paper introduces the Psychological Reg…
Reinforcement LearningContinuous ControlReinforcement Learning from Delayed Observations via World Models
In standard reinforcement learning settings, agents typically assume immediate feedback about the effects of their actions after taking them. However, in practice, this assumption may not hold true due to physical constr…
continuous-controlContinuous Controlreinforcement-learningReinforcement LearningBoosting Reinforcement Learning with Strongly Delayed Feedback Through Auxiliary Short Delays
Reinforcement learning (RL) is challenging in the common case of delays between events and their sensory perceptions. State-of-the-art (SOTA) state augmentation techniques either suffer from state space explosion or perf…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)