Why Do Animals Need Shaping? A Theory of Task Composition and Curriculum Learning
Diverse studies in systems neuroscience begin with extended periods of curriculum training known as `shaping' procedures. These involve progressively studying component parts of more complex tasks, and can make the difference between learning a task quickly, slowly or not at all. Despite the importance of shaping to the acquisition of complex tasks, there is as yet no theory that can help guide the design of shaping procedures, or more fundamentally, provide insight into its key role in learning. Modern deep reinforcement learning systems might implicitly learn compositional primitives within their multilayer policy networks. Inspired by these models, we propose and analyse a model of deep policy gradient learning of simple compositional reinforcement learning tasks. Using the tools of statistical physics, we solve for exact learning dynamics and characterise different learning strategies including primitives pre-training, in which task primitives are studied individually before learning compositional tasks. We find a complex interplay between task complexity and the efficacy of shaping strategies. Overall, our theory provides an analytical understanding of the benefits of shaping in a class of compositional tasks and a quantitative account of how training protocols can disclose useful task primitives, ultimately yielding faster and more robust learning.
Code (0)
등록된 구현이 없습니다.
Tasks
Deep Reinforcement Learningreinforcement-learningReinforcement LearningSimilar Papers 제목 키워드 기반
Simulating how animals learn: a new modelling framework applied to the process of optimal foraging
Animal learning has interested ecologists and psychologists for over a century. Mathematical models that explain how animals store and recall information have gained attention recently. Central to this work is statistica…
Bayesian InferenceUtilizing Prior Solutions for Reward Shaping and Composition in Entropy-Regularized Reinforcement Learning
In reinforcement learning (RL), the ability to utilize prior knowledge from previously solved tasks can allow agents to quickly solve new problems. In some cases, these new problems may be approximately solved by composi…
reinforcement-learningReinforcement Learning (RL)RelationFlexible task abstractions emerge in linear networks with fast and bounded units
Animals survive in dynamic environments changing at arbitrary timescales, but such data distribution shifts are a challenge to neural networks. To adapt to change, neural systems may change a large number of parameters, …
Metabolic scaling is governed by Murray's network in animals and by hydraulic conductance and photosynthesis in plants
The prevailing theory for metabolic scaling is based on area-preserved, space-filling fractal vascular networks. However, it's known both theoretically and experimentally that animals' vascular systems obey Murray's cubi…
Semantic Part Segmentation using Compositional Model combining Shape and Appearance
In this paper, we study the problem of semantic part segmentation for animals. This is more challenging than standard object detection, object segmentation and pose estimation tasks because semantic parts of animals ofte…
Objectobject-detectionObject DetectionPose Estimation+2