Reinforcement Learning with Structured Hierarchical Grammar Representations of Actions

2019-10-07 · Petros Christodoulou, Robert Tjarko Lange, Ali Shafti, A. Aldo Faisal

From a young age humans learn to use grammatical principles to hierarchically combine words into sentences. Action grammars is the parallel idea, that there is an underlying set of rules (a "grammar") that govern how we hierarchically combine actions to form new, more complex actions. We introduce the Action Grammar Reinforcement Learning (AG-RL) framework which leverages the concept of action grammars to consistently improve the sample efficiency of Reinforcement Learning agents. AG-RL works by using a grammar inference algorithm to infer the "action grammar" of an agent midway through training. The agent's action space is then augmented with macro-actions identified by the grammar. We apply this framework to Double Deep Q-Learning (AG-DDQN) and a discrete action version of Soft Actor-Critic (AG-SAC) and find that it improves performance in 8 out of 8 tested Atari games (median +31%, max +668%) and 19 out of 20 tested Atari games (median +96%, maximum +3,756%) respectively without substantive hyperparameter tuning. We also show that AG-SAC beats the model-free state-of-the-art for sample efficiency in 17 out of the 20 tested Atari games (median +62%, maximum +13,140%), again without substantive hyperparameter tuning.

📄 PDF Abstract BibTeX arXiv:1910.02876

Code (0)

등록된 구현이 없습니다.

Tasks

Atari GamesQ-Learningreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

Exploring the hierarchical structure of human plans via program generation

2023-11-30 · Carlos G. Correa, Sophia Sanborn, Mark K. Ho, Frederick Callaway 외

Human behavior is often assumed to be hierarchically structured, made up of abstract actions that can be decomposed into concrete actions. However, behavior is typically measured as a sequence of actions, which makes it …

Semantic RL with Action Grammars: Data-Efficient Learning of Hierarchical Task Abstractions

2019-07-29 · Robert Tjarko Lange, Aldo Faisal

Hierarchical Reinforcement Learning algorithms have successfully been applied to temporal credit assignment problems with sparse reward signals. However, state-of-the-art algorithms require manual specification of sub-ta…

Hierarchical Reinforcement Learning Logical Reasoning reinforcement-learning Reinforcement Learning+2

Hierarchical Reinforcement Learning Program induction Reinforcement Learning Reinforcement Learning (RL)

Reinforcement Learning with Structured Hierarchical Grammar Representations of Actions

Code (0)

Tasks

Methods 이 논문이 사용한 방법론

Similar Papers 제목 키워드 기반

Exploring the hierarchical structure of human plans via program generation

Semantic RL with Action Grammars: Data-Efficient Learning of Hierarchical Task Abstractions

Disjoint Processing Mechanisms of Hierarchical and Linear Grammars in Large Language Models

ChartMark: A Structured Grammar for Chart Annotation

Representational efficiency outweighs action efficiency in human program induction