ODE-based Recurrent Model-free Reinforcement Learning for POMDPs

2023-09-25 · NeurIPS 2023 11

Neural ordinary differential equations (ODEs) are widely recognized as the standard for modeling physical mechanisms, which help to perform approximate inference in unknown physical or biological environments. In partially observable (PO) environments, how to infer unseen information from raw observations puzzled the agents. By using a recurrent policy with a compact context, context-based reinforcement learning provides a flexible way to extract unobservable information from historical transitions. To help the agent extract more dynamics-related information, we present a novel ODE-based recurrent model combines with model-free reinforcement learning (RL) framework to solve partially observable Markov decision processes (POMDPs). We experimentally demonstrate the efficacy of our methods across various PO continuous control and meta-RL tasks. Furthermore, our experiments illustrate that our method is robust against irregular observations, owing to the ability of ODEs to model irregularly-sampled time series.

📄 PDF Abstract BibTeX arXiv:2309.14078

Code (0)

등록된 구현이 없습니다.

Tasks

continuous-control Continuous Control reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Time Series

Methods 이 논문이 사용한 방법론

PO Stochastic optimization methods have gained significant prominence as effective techniques in contemporary research, addressing complex optimization challenges efficiently. This…

Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPs

2021-10-11 · Tianwei Ni, Benjamin Eysenbach, Ruslan Salakhutdinov

Many problems in RL, such as meta-RL, robust RL, generalization in RL, and temporal credit assignment, can be cast as POMDPs. In theory, simply augmenting model-free RL with memory-based architectures, such as recurrent …

Finite-State Controllers for (Hidden-Model) POMDPs using Deep Reinforcement Learning

2026-02-09 · David Hudák, Maris F. L. Galesloot, Martin Tappler, Martin Kurečka 외 arxiv

Solving partially observable Markov decision processes (POMDPs) requires computing policies under imperfect state information. Despite recent advances, the scalability of existing POMDP solvers remains limited. Moreover,…

Reinforcement Learning

Reinforcement Learning in POMDPs with Memoryless Options and Option-Observation Initiation Sets

2017-08-22 · Denis Steckelmacher, Diederik M. Roijers, Anna Harutyunyan, Peter Vrancx 외

Many real-world reinforcement learning problems have a hierarchical nature, and often exhibit some degree of partial observability. While hierarchy and partial observability are usually tackled separately (for instance b…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Periodic agent-state based Q-learning for POMDPs

2024-07-08 · Amit Sinha, Matthieu Geist, Aditya Mahajan

The standard approach for Partially Observable Markov Decision Processes (POMDPs) is to convert them to a fully observed belief-state MDP. However, the belief state depends on the system model and is therefore not viable…

Q-Learning Reinforcement Learning (RL)

Rethinking Transformers in Solving POMDPs

2024-05-27 · Chenhao Lu, Ruizhe Shi, Yuyao Liu, Kaizhe Hu 외

Sequential decision-making algorithms such as reinforcement learning (RL) in real-world scenarios inevitably face environments with partial observability. This paper scrutinizes the effectiveness of a popular architectur…

Decision Making Reinforcement Learning (RL)Sequential Decision Making