Continuously Discovering Novel Strategies via Reward-Switching Policy Optimization
We present Reward-Switching Policy Optimization (RSPO), a paradigm to discover diverse strategies in complex RL environments by iteratively finding novel policies that are both locally optimal and sufficiently different from existing ones. To encourage the learning policy to consistently converge towards a previously undiscovered local optimum, RSPO switches between extrinsic and intrinsic rewards via a trajectory-based novelty measurement during the optimization process. When a sampled trajectory is sufficiently distinct, RSPO performs standard policy optimization with extrinsic rewards. For trajectories with high likelihood under existing policies, RSPO utilizes an intrinsic diversity reward to promote exploration. Experiments show that RSPO is able to discover a wide spectrum of strategies in a variety of domains, ranging from single-agent particle-world tasks and MuJoCo continuous control to multi-agent stag-hunt games and StarCraftII challenges.
Code (0)
등록된 구현이 없습니다.
Tasks
continuous-controlContinuous ControlDiversityMuJoCoSimilar Papers 제목 키워드 기반
Discovering Diverse Multi-Agent Strategic Behavior via Reward Randomization
We propose a simple, general and effective technique, Reward Randomization for discovering diverse strategic policies in complex multi-agent games. Combining reward randomization and policy gradient, we derive a new algo…
DGPO: Discovering Multiple Strategies with Diversity-Guided Policy Optimization
Most reinforcement learning algorithms seek a single optimal strategy that solves a given task. However, it can often be valuable to learn a diverse set of solutions, for instance, to make an agent's interaction with use…
Diversityreinforcement-learningReinforcement LearningReinforcement Learning (RL)Program-Based Strategy Induction for Reinforcement Learning
Typical models of learning assume incremental estimation of continuously-varying decision variables like expected rewards. However, this class of models fails to capture more idiosyncratic, discrete heuristics and strate…
Incremental LearningProgram inductionreinforcement-learningReinforcement LearningHMM for Discovering Decision-Making Dynamics Using Reinforcement Learning Experiments
Major depressive disorder (MDD) presents challenges in diagnosis and treatment due to its complex and heterogeneous nature. Emerging evidence indicates that reward processing abnormalities may serve as a behavioral marke…
Decision Makingparameter estimationreinforcement-learningReinforcement Learning+1Online Caching with Optimal Switching Regret
We consider the classical uncoded caching problem from an online learning point-of-view. A cache of limited storage capacity can hold $C$ files at a time from a large catalog. A user requests an arbitrary file from the c…