Reinforcement Learning for Control with Multiple Frequencies

2020-12-01 · NeurIPS 2020 12 · Jongmin Lee, ByungJun Lee, Kee-Eung Kim

Many real-world sequential decision problems involve multiple action variables whose control frequencies are different, such that actions take their effects at different periods. While these problems can be formulated with the notion of multiple action persistences in factored-action MDP (FA-MDP), it is non-trivial to solve them efficiently since an action-persistent policy constructed from a stationary policy can be arbitrarily suboptimal, rendering solution methods for the standard FA-MDPs hardly applicable. In this paper, we formalize the problem of multiple control frequencies in RL and provide its efficient solution method. Our proposed method, Action-Persistent Policy Iteration (AP-PI), provides a theoretical guarantee on the convergence to an optimal solution while incurring only a factor of $|A|$ increase in time complexity during policy improvement step, compared to the standard policy iteration for FA-MDPs. Extending this result, we present Action-Persistent Actor-Critic (AP-AC), a scalable RL algorithm for high-dimensional control tasks. In the experiments, we demonstrate that AP-AC significantly outperforms the baselines on several continuous control tasks and a traffic control simulation, which highlights the effectiveness of our method that directly optimizes the periodic non-stationary policy for tasks with multiple control frequencies.

📄 PDF Abstract BibTeX

Code (0)

등록된 구현이 없습니다.

Tasks

continuous-control Continuous Control reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Offline Reinforcement Learning at Multiple Frequencies

2022-07-26 · Kaylee Burns, Tianhe Yu, Chelsea Finn, Karol Hausman

Leveraging many sources of offline robot data requires grappling with the heterogeneity of such data. In this paper, we focus on one particular aspect of heterogeneity: learning from offline data collected at different c…

Offline RL reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Overcoming Slow Decision Frequencies in Continuous Control: Model-Based Sequence Reinforcement Learning for Model-Free Control

2024-10-11 · Devdhar Patel, Hava Siegelmann

Reinforcement learning (RL) is rapidly reaching and surpassing human-level control capabilities. However, state-of-the-art RL algorithms often require timesteps and reaction times significantly faster than human capabili…

continuous-control Continuous Control model Reinforcement Learning (RL)

Real Time Control of Tandem-Wing Experimental Platform Using Concerto Reinforcement Learning

2025-02-08 · Zhang Minghao, Yang Xiaojun, Wang Zhihe, Wang Liang

This paper introduces the CRL2RT algorithm, an advanced reinforcement learning method aimed at improving the real-time control performance of the Direct-Drive Tandem-Wing Experimental Platform (DDTWEP). Inspired by drago…

reinforcement-learning Reinforcement Learning

Hierarchical Decision Transformer

2022-09-21 · André Correia, Luís A. Alexandre

Sequence models in reinforcement learning require task knowledge to estimate the task policy. This paper presents a hierarchical algorithm for learning a sequence model from demonstrations. The high-level mechanism guide…

D4RL reinforcement-learning Reinforcement Learning (RL)

ConcertoRL: An Innovative Time-Interleaved Reinforcement Learning Approach for Enhanced Control in Direct-Drive Tandem-Wing Vehicles

2024-05-22 · Minghao Zhang, Bifeng Song, Changhao Chen, Xinyu Lang

In control problems for insect-scale direct-drive experimental platforms under tandem wing influence, the primary challenge facing existing reinforcement learning models is their limited safety in the exploration process…