Decentralized Cross-Entropy Method for Model-Based Reinforcement Learning
Cross-Entropy Method (CEM) is a popular approach to planning in model-based reinforcement learning. It has so far always taken a \textit{centralized} approach where the sampling distribution is updated \textit{centrally} based on the result of a top-$k$ operation applied to \textit{all samples}. We show that such a \textit{centralized} approach makes CEM vulnerable to local optima and impair its sample efficiency, even in a one-dimensional multi-modal optimization task. In this paper, we propose \textbf{Decent}ralized \textbf{CEM (DecentCEM)} where an ensemble of CEM instances run independently from one another and each performs a local improvement of its own sampling distribution. In the exemplar optimization task, the proposed decentralized approach DecentCEM finds the global optimum much more consistently than the existing CEM approaches that use either a single Gaussian distribution or a mixture of Gaussians. Further, we extend the decentralized approach to sequential decision-making problems where we show in 13 continuous control benchmark environments that it matches or outperforms the state-of-the-art CEM algorithms in most cases, under the same budget of the total number of samples for planning.
Code (0)
등록된 구현이 없습니다.
Tasks
continuous-controlContinuous ControlDecision MakingModel-based Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)Sequential Decision MakingSimilar Papers 제목 키워드 기반
Multi-Agent Cross-Entropy Method with Monotonic Nonlinear Critic Decomposition
Cooperative multi-agent reinforcement learning (MARL) commonly adopts centralized training with decentralized execution (CTDE), where centralized critics leverage global information to guide decentralized actors. However…
Multi-agent Reinforcement LearningSoft-QMIX: Integrating Maximum Entropy For Monotonic Value Function Factorization
Multi-agent reinforcement learning (MARL) tasks often utilize a centralized training with decentralized execution (CTDE) framework. QMIX is a successful CTDE method that learns a credit assignment function to derive loca…
Multi-agent Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)+2A Simple Decentralized Cross-Entropy Method
Cross-Entropy Method (CEM) is commonly used for planning in model-based reinforcement learning (MBRL) where a centralized approach is typically utilized to update the sampling distribution based on only the top-$k$ opera…
continuous-controlContinuous ControlModel-based Reinforcement LearningDiffusing to Coordinate: Efficient Online Multi-Agent Diffusion Policies
Online Multi-Agent Reinforcement Learning (MARL) is a prominent framework for efficient agent coordination. Crucially, enhancing policy expressiveness is pivotal for achieving superior performance. Diffusion-based genera…
Multi-agent Reinforcement LearningImage GenerationA Decentralized Policy Gradient Approach to Multi-task Reinforcement Learning
We develop a mathematical framework for solving multi-task reinforcement learning (MTRL) problems based on a type of policy gradient method. The goal in MTRL is to learn a common policy that operates effectively in diffe…
Atari GamesMulti-Task LearningNavigatereinforcement-learning+2