Self-Imitation Advantage Learning
Self-imitation learning is a Reinforcement Learning (RL) method that encourages actions whose returns were higher than expected, which helps in hard exploration and sparse reward problems. It was shown to improve the performance of on-policy actor-critic methods in several discrete control tasks. Nevertheless, applying self-imitation to the mostly action-value based off-policy RL methods is not straightforward. We propose SAIL, a novel generalization of self-imitation learning for off-policy RL, based on a modification of the Bellman optimality operator that we connect to Advantage Learning. Crucially, our method mitigates the problem of stale returns by choosing the most optimistic return estimate between the observed return and the current action-value for self-imitation. We demonstrate the empirical effectiveness of SAIL on the Arcade Learning Environment, with a focus on hard exploration games.
Code (0)
등록된 구현이 없습니다.
Tasks
Atari GamesImitation LearningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Self-Imitation Learning
This paper proposes Self-Imitation Learning (SIL), a simple off-policy actor-critic algorithm that learns to reproduce the agent's past good decisions. This algorithm is designed to verify our hypothesis that exploiting …
Atari GamesImitation LearningMuJoCoImitation Learning for Generalizable Self-driving Policy with Sim-to-real Transfer
Imitation Learning uses the demonstrations of an expert to uncover the optimal policy and it is suitable for real-world robotics tasks as well. In this case, however, the training of the agent is carried out in a simulat…
Imitation LearningTransfer LearningTriplets Better Than Pairs: Towards Stable and Effective Self-Play Fine-Tuning for LLMs
Recently, self-play fine-tuning (SPIN) has been proposed to adapt large language models to downstream applications with scarce expert-annotated data, by iteratively generating synthetic responses from the model itself. H…
Improving Sequential Recommendation Consistency with Self-Supervised Imitation
Most sequential recommendation models capture the features of consecutive items in a user-item interaction history. Though effective, their representation expressiveness is still hindered by the sparse learning signals. …
Imitation LearningSequential RecommendationSparse LearningPotential-Guided Flow Matching for Vision-Language-Action Policy Improvement
Large vision-language-action (VLA) policies are increasingly trained as conditional generative models over action chunks. Yet deployment produces mixed-quality experience-successful demonstrations, partial completions, r…
Reinforcement Learning