Learning Sparse Representations Incrementally in Deep Reinforcement Learning

2019-12-09 · J. Fernando Hernandez-Garcia, Richard S. Sutton

Sparse representations have been shown to be useful in deep reinforcement learning for mitigating catastrophic interference and improving the performance of agents in terms of cumulative reward. Previous results were based on a two step process were the representation was learned offline and the action-value function was learned online afterwards. In this paper, we investigate if it is possible to learn a sparse representation and the action-value function simultaneously and incrementally. We investigate this question by employing several regularization techniques and observing how they affect sparsity of the representation learned by a DQN agent in two different benchmark domains. Our results show that with appropriate regularization it is possible to increase the sparsity of the representations learned by DQN agents. Moreover, we found that learning sparse representations also resulted in improved performance in terms of cumulative reward. Finally, we found that the performance of the agents that learned a sparse representation was more robust to the size of the experience replay buffer. This last finding supports the long standing hypothesis that the overlap in representations learned by deep neural networks is the leading cause of catastrophic interference.

📄 PDF Abstract BibTeX arXiv:1912.04002

Code (0)

등록된 구현이 없습니다.

Tasks

Deep Reinforcement Learningreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

Experience Replay Experience Replay is a replay memory technique used in reinforcement learning where we store the agent’s experiences at each time-step, $e\_{t} = \left(s\_{t}, a\_{t}, r\_{t},…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Convolution A convolution is a type of matrix operation, consisting of a kernel, a small matrix of weights, that slides over input data performing element-wise multiplication with the…

DQN A DQN, or Deep Q-Network, approximates a state-value function in a Q-Learning framework with a neural network. In the Atari…

The Utility of Sparse Representations for Control in Reinforcement Learning

2018-11-15 · Vincent Liu, Raksha Kumaraswamy, Lei Le, Martha White

We investigate sparse representations for control in reinforcement learning. While these representations are widely used in computer vision, their prevalence in reinforcement learning is limited to sparse coding where ex…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Learning Sparse Representations in Reinforcement Learning with Sparse Coding

2017-07-26 · Lei Le, Raksha Kumaraswamy, Martha White

A variety of representation learning approaches have been investigated for reinforcement learning; much less attention, however, has been given to investigating the utility of sparse coding. Outside of reinforcement lear…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Representation Learning

Go ahead and do not forget: Modular lifelong learning from event-based data

2022-08-21 · Neurocomputing 2022 8 · VadymGryshchuk, CorneliusWeber, Chu KiongLoo, StefanWermter

Lifelong learning is a long-standing aim for artificial agents that act in dynamic environments in which an agent needs to accumulate knowledge incrementally without forgetting previously learned representations. Contemp…

Incremental Learning Lifelong learning

Wield: Systematic Reinforcement Learning With Progressive Randomization

2019-09-15 · Michael Schaarschmidt, Kai Fricke, Eiko Yoneki

Reinforcement learning frameworks have introduced abstractions to implement and execute algorithms at scale. They assume standardized simulator interfaces but are not concerned with identifying suitable task representati…

General Classification reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Sparse Meta Networks for Sequential Adaptation and its Application to Adaptive Language Modelling

2020-09-03 · Tsendsuren Munkhdalai

Training a deep neural network requires a large amount of single-task data and involves a long time-consuming optimization phase. This is not scalable to complex, realistic environments with new unexpected changes. Human…

Incremental Learning Inductive Bias Language Modelling Meta-Learning+1