Efficient Exploration via State Marginal Matching
Exploration is critical to a reinforcement learning agent's performance in its given environment. Prior exploration methods are often based on using heuristic auxiliary predictions to guide policy behavior, lacking a mathematically-grounded objective with clear properties. In contrast, we recast exploration as a problem of State Marginal Matching (SMM), where we aim to learn a policy for which the state marginal distribution matches a given target state distribution. The target distribution is a uniform distribution in most cases, but can incorporate prior knowledge if available. In effect, SMM amortizes the cost of learning to explore in a given environment. The SMM objective can be viewed as a two-player, zero-sum game between a state density model and a parametric policy, an idea that we use to build an algorithm for optimizing the SMM objective. Using this formalism, we further demonstrate that prior work approximately maximizes the SMM objective, offering an explanation for the success of these methods. On both simulated and real-world tasks, we demonstrate that agents that directly optimize the SMM objective explore faster and adapt more quickly to new tasks as compared to prior exploration methods.
Code (1)
Tasks
Efficient ExplorationReinforcement LearningUnsupervised Reinforcement LearningSimilar Papers 제목 키워드 기반
From Relative Entropy to Minimax: A Unified Framework for Coverage in MDPs
Targeted and deliberate exploration of state--action pairs is essential in reward-free Markov Decision Problems (MDPs). More precisely, different state-action pairs exhibit different degree of importance or difficulty wh…
Latent State Marginalization as a Low-cost Approach for Improving Exploration
While the maximum entropy (MaxEnt) reinforcement learning (RL) framework -- often touted for its exploration and robustness capabilities -- is usually motivated from a probabilistic perspective, the use of deep probabili…
continuous-controlContinuous ControlReinforcement Learning (RL)SMAC+1Joint Probabilistic Matching Using m-Best Solutions
Matching between two sets of objects is typically approached by finding the object pairs that collectively maximize the joint matching score. In this paper, we argue that this single solution does not necessarily lead to…
Person Re-IdentificationMarginalized State Distribution Entropy Regularization in Policy Optimization
Entropy regularization is used to get improved optimization performance in reinforcement learning tasks. A common form of regularization is to maximize policy entropy to avoid premature convergence and lead to more stoch…
continuous-controlContinuous ControlReinforcement LearningPreference-Guided Reinforcement Learning for Efficient Exploration
In this paper, we investigate preference-based reinforcement learning (PbRL) that allows reinforcement learning (RL) agents to learn from human feedback. This is particularly valuable when defining a fine-grain reward fu…
Efficient Explorationreinforcement-learningReinforcement LearningReinforcement Learning (RL)