Distributed Algorithms for Linearly-Solvable Optimal Control in Networked Multi-Agent Systems
Distributed algorithms for both discrete-time and continuous-time linearly solvable optimal control (LSOC) problems of networked multi-agent systems (MASs) are investigated in this paper. A distributed framework is proposed to partition the optimal control problem of a networked MAS into several local optimal control problems in factorial subsystems, such that each (central) agent behaves optimally to minimize the joint cost function of a subsystem that comprises a central agent and its neighboring agents, and the local control actions (policies) only rely on the knowledge of local observations. Under this framework, we not only preserve the correlations between neighboring agents, but moderate the communication and computational complexities by decentralizing the sampling and computational processes over the network. For discrete-time systems modeled by Markov decision processes, the joint Bellman equation of each subsystem is transformed into a system of linear equations and solved using parallel programming. For continuous-time systems modeled by It\^o diffusion processes, the joint optimality equation of each subsystem is converted into a linear partial differential equation, whose solution is approximated by a path integral formulation and a sample-efficient relative entropy policy search algorithm, respectively. The learned control policies are generalized to solve the unlearned tasks by resorting to the compositionality principle, and illustrative examples of cooperative UAV teams are provided to verify the effectiveness and advantages of these algorithms.
Code (0)
등록된 구현이 없습니다.
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Compositionality of Linearly Solvable Optimal Control in Networked Multi-Agent Systems
In this paper, we discuss the methodology of generalizing the optimal control law from learned component tasks to unlearned composite tasks on Multi-Agent Systems (MASs), by using the linearity composition principle of l…
Tsallis Entropy Regularization for Linearly Solvable MDP and Linear Quadratic Regulator
Shannon entropy regularization is widely adopted in optimal control due to its ability to promote exploration and enhance robustness, e.g., maximum entropy reinforcement learning known as Soft Actor-Critic. In this paper…
reinforcement-learningReinforcement LearningHierarchy through Composition with Linearly Solvable Markov Decision Processes
Hierarchical architectures are critical to the scalability of reinforcement learning methods. Current hierarchical frameworks execute actions serially, with macro-actions comprising sequences of primitive actions. We pro…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Data-Driven Learning and Load Ensemble Control
Demand response (DR) programs aim to engage distributed small-scale flexible loads, such as thermostatically controllable loads (TCLs), to provide various grid support services. Linearly Solvable Markov Decision Process …
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Privacy-Aware Load Ensemble Control: A Linearly-Solvable MDP Approach
Demand response (DR) programs engage distributed demand-side resources, e.g., controllable residential and commercial loads, in providing ancillary services for electric power systems. Ensembles of these resources can he…
Privacy Preserving