Meta-Reinforcement Learning for the Tuning of PI Controllers: An Offline Approach
Meta-learning is a branch of machine learning which trains neural network models to synthesize a wide variety of data in order to rapidly solve new problems. In process control, many systems have similar and well-understood dynamics, which suggests it is feasible to create a generalizable controller through meta-learning. In this work, we formulate a meta reinforcement learning (meta-RL) control strategy that can be used to tune proportional--integral controllers. Our meta-RL agent has a recurrent structure that accumulates "context" to learn a system's dynamics through a hidden state variable in closed-loop. This architecture enables the agent to automatically adapt to changes in the process dynamics. In tests reported here, the meta-RL agent was trained entirely offline on first order plus time delay systems, and produced excellent results on novel systems drawn from the same distribution of process dynamics used for training. A key design element is the ability to leverage model-based information offline during training in simulated environments while maintaining a model-free policy structure for interacting with novel processes where there is uncertainty regarding the true process dynamics. Meta-learning is a promising approach for constructing sample-efficient intelligent controllers.
Code (0)
등록된 구현이 없습니다.
Tasks
Meta-LearningMeta Reinforcement Learningreinforcement-learningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Meta-Reinforcement Learning for Adaptive Control of Second Order Systems
Meta-learning is a branch of machine learning which aims to synthesize data from a distribution of related tasks to efficiently solve new ones. In process control, many systems have similar and well-understood dynamics, …
Meta-LearningMeta Reinforcement Learningreinforcement-learningReinforcement Learning+1Contextual Transformer for Offline Meta Reinforcement Learning
The pretrain-finetuning paradigm in large-scale sequence models has made significant progress in natural language processing and computer vision tasks. However, such a paradigm is still hindered by several challenges in …
D4RLMeta Reinforcement LearningMuJoCoOffline RL+5Offline Meta-Reinforcement Learning with Advantage Weighting
This paper introduces the offline meta-reinforcement learning (offline meta-RL) problem setting and proposes an algorithm that performs well in this setting. Offline meta-RL is analogous to the widely successful supervis…
Machine TranslationMeta-LearningMeta Reinforcement LearningObject Recognition+4Whole-Body Mobile Manipulation using Offline Reinforcement Learning on Sub-optimal Controllers
Mobile Manipulation (MoMa) of articulated objects, such as opening doors, drawers, and cupboards, demands simultaneous, whole-body coordination between a robot's base and arms. Classical whole-body controllers (WBCs) can…
Reinforcement LearningOffline RLOffline Hierarchical Reinforcement Learning via Inverse Optimization
Hierarchical policies enable strong performance in many sequential decision-making problems, such as those with high-dimensional action spaces, those requiring long-horizon planning, and settings with sparse rewards. How…
Decision MakingHierarchical Reinforcement Learningreinforcement-learningReinforcement Learning+2