Multi-timestep models for Model-based Reinforcement Learning

2023-10-09 · Abdelhakim Benechehab, Giuseppe Paolo, Albert Thomas, Maurizio Filippone, Balázs Kégl

In model-based reinforcement learning (MBRL), most algorithms rely on simulating trajectories from one-step dynamics models learned on data. A critical challenge of this approach is the compounding of one-step prediction errors as length of the trajectory grows. In this paper we tackle this issue by using a multi-timestep objective to train one-step models. Our objective is a weighted sum of a loss function (e.g., negative log-likelihood) at various future horizons. We explore and test a range of weights profiles. We find that exponentially decaying weights lead to models that significantly improve the long-horizon R2 score. This improvement is particularly noticeable when the models were evaluated on noisy data. Finally, using a soft actor-critic (SAC) agent in pure batch reinforcement learning (RL) and iterated batch RL scenarios, we found that our multi-timestep models outperform or match standard one-step models. This was especially evident in a noisy variant of the considered environment, highlighting the potential of our approach in real-world applications.

📄 PDF Abstract BibTeX arXiv:2310.05672

Code (0)

등록된 구현이 없습니다.

Tasks

modelModel-based Reinforcement Learningreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Sparse Adversarial Attack in Multi-agent Reinforcement Learning

2022-05-19 · Yizheng Hu, Zhihua Zhang

Cooperative multi-agent reinforcement learning (cMARL) has many real applications, but the policy trained by existing cMARL algorithms is not robust enough when deployed. There exist also many methods about adversarial a…

Adversarial Attack Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning+1

Reinforcement Learning from Denoising Feedback

2026-05-25 · Qi He, Huan Chen, Ya Guo, Huijia Zhu 외 arxiv

Policy loss estimation remains a fundamental and long-standing challenge in reinforcement learning (RL) for diffusion language models (DLMs). We introduce Reinforcement Learning from Denoising Feedback (RLDF), a novel tr…

Computational Efficiency Reinforcement Learning

Communication Gain and Delay Cost Under Cross-Timestep Delays in Cooperative Multi-Agent Reinforcement Learning

2026-04-04 · Zihong Gao, Hongjian Liang, Lei Hao, Liangjun Ke arxiv

Communication is essential for coordination in \emph{cooperative} multi-agent reinforcement learning under partial observability, yet \emph{cross-timestep} delays cause messages to arrive multiple timesteps after generat…

Multi-agent Reinforcement Learning

SEENN: Towards Temporal Spiking Early-Exit Neural Networks

2023-04-02 · Yuhang Li, Tamar Geller, Youngeun Kim, Priyadarshini Panda

Spiking Neural Networks (SNNs) have recently become more popular as a biologically plausible substitute for traditional Artificial Neural Networks (ANNs). SNNs are cost-efficient and deployment-friendly because they proc…

SEENN: Towards Temporal Spiking Early Exit Neural Networks

2023-09-21 · NeurIPS 2023 11