Recurrent Reinforcement Learning with Memoroids
Memory models such as Recurrent Neural Networks (RNNs) and Transformers address Partially Observable Markov Decision Processes (POMDPs) by mapping trajectories to latent Markov states. Neither model scales particularly well to long sequences, especially compared to an emerging class of memory models called Linear Recurrent Models. We discover that the recurrent update of these models resembles a monoid, leading us to reformulate existing models using a novel monoid-based framework that we call memoroids. We revisit the traditional approach to batching in recurrent reinforcement learning, highlighting theoretical and empirical deficiencies. We leverage memoroids to propose a batching method that improves sample efficiency, increases the return, and simplifies the implementation of recurrent loss functions in reinforcement learning.
Code (1)
Tasks
reinforcement-learningReinforcement LearningSimilar Papers 제목 키워드 기반
Investigating Action Encodings in Recurrent Neural Networks in Reinforcement Learning
Building and maintaining state to learn policies and value functions is critical for deploying reinforcement learning (RL) agents in the real world. Recurrent neural networks (RNNs) have become a key point of interest fo…
Reinforcement LearningRecurrent Off-Policy Deep Reinforcement Learning Doesn't Have to be Slow
Recurrent off-policy deep reinforcement learning models achieve state-of-the-art performance but are often sidelined due to their high computational demands. In response, we introduce RISE (Recurrent Integration via Simp…
Reinforcement LearningReal-Time Recurrent Reinforcement Learning
We introduce a biologically plausible RL framework for solving tasks in partially observable Markov decision processes (POMDPs). The proposed algorithm combines three integral parts: (1) A Meta-RL architecture, resemblin…
continuous-controlContinuous ControlPartially Observable Reinforcement Learningreinforcement-learning+1Performing Deep Recurrent Double Q-Learning for Atari Games
Currently, many applications in Machine Learning are based on define new models to extract more information about data, In this case Deep Reinforcement Learning with the most common application in video games like Atari,…
Atari GamesDeep Reinforcement LearningQ-Learningreinforcement-learning+2Reinforcement Learning with Low-Complexity Liquid State Machines
We propose reinforcement learning on simple networks consisting of random connections of spiking neurons (both recurrent and feed-forward) that can learn complex tasks with very little trainable parameters. Such sparse a…
Atari GamesDeep Reinforcement LearningQ-Learningreinforcement-learning+2