DIRECT: Learning from Sparse and Shifting Rewards using Discriminative Reward Co-Training
We propose discriminative reward co-training (DIRECT) as an extension to deep reinforcement learning algorithms. Building upon the concept of self-imitation learning (SIL), we introduce an imitation buffer to store beneficial trajectories generated by the policy determined by their return. A discriminator network is trained concurrently to the policy to distinguish between trajectories generated by the current policy and beneficial trajectories generated by previous policies. The discriminator's verdict is used to construct a reward signal for optimizing the policy. By interpolating prior experience, DIRECT is able to act as a surrogate, steering policy optimization towards more valuable regions of the reward landscape thus learning an optimal policy. Our results show that DIRECT outperforms state-of-the-art algorithms in sparse- and shifting-reward environments being able to provide a surrogate reward to the policy and direct the optimization towards valuable areas.
Code (0)
등록된 구현이 없습니다.
Tasks
Deep Reinforcement LearningImitation LearningSimilar Papers 제목 키워드 기반
TRL: Discriminative Hints for Scalable Reverse Curriculum Learning
Deep reinforcement learning algorithms have proven successful in a variety of domains. However, tasks with sparse rewards remain challenging when the state space is large. Goal-oriented tasks are among the most typical p…
Deep Reinforcement LearningReinforcement LearningRobot ManipulationDelTA: Discriminative Token Credit Assignment for Reinforcement Learning from Verifiable Rewards
Reinforcement learning from verifiable rewards (RLVR) has emerged as a central technique for improving the reasoning capabilities of large language models. Despite its effectiveness, how response-level rewards translate …
Reinforcement LearningCode GenerationMulti-Turn Reinforcement Learning for Tool-Calling Agents with Iterative Reward Calibration
Training tool-calling agents with reinforcement learning on multi-turn tasks remains challenging due to sparse outcome rewards and difficult credit assignment across conversation turns. We present the first application o…
Reinforcement LearningDEIR: Efficient and Robust Exploration through Discriminative-Model-Based Episodic Intrinsic Rewards
Exploration is a fundamental aspect of reinforcement learning (RL), and its effectiveness is a deciding factor in the performance of RL algorithms, especially when facing sparse extrinsic rewards. Recent studies have sho…
Reinforcement Learning (RL)Back on Track: Aligning Rewards and States for Reasoning in Diffusion Large Language Models
Reinforcement learning (RL) holds immense promise for enhancing the reasoning capabilities of diffusion large language models (dLLMs). However, progress is fundamentally constrained by a dual misalignment between authent…
Reinforcement Learning