Learning Adaptive Control Flow in Transformers for Improved Systematic Generalization
Despite successes across a broad range of applications, Transformers have limited capability in systematic generalization. The situation is especially frustrating for algorithmic tasks, where they often fail to find intuitive solutions that can be simply expressed in terms of attention patterns. Here we propose two modifications to the Transformer architecture, copy gate and geometric attention, which facilitate learning such intuitive and interpretable solutions to algorithmic problems. Our novel Transformer, called Transformer Control Flow (TCF) achieves 100% length generalization accuracy on the classic compositional table lookup task. The resulting attention and gating patterns are interpretable, demonstrating that the model implements adaptive control flow.
Code (0)
등록된 구현이 없습니다.
Tasks
Systematic GeneralizationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Adaptive Control Flow in Transformers Improves Systematic Generalization
Despite successes across a broad range of applications, Transformers have limited capability in systematic generalization. The situation is especially frustrating in the case of algorithmic tasks, where they often fail t…
ListOpsSystematic GeneralizationThe Neural Data Router: Adaptive Control Flow in Transformers Improves Systematic Generalization
Despite progress across a broad range of applications, Transformers have limited success in systematic generalization. The situation is especially frustrating in the case of algorithmic tasks, where they often fail to fi…
ListOpsSystematic GeneralizationMitigating Partial Observability in Adaptive Traffic Signal Control with Transformers
Efficient traffic signal control is essential for managing urban transportation, minimizing congestion, and improving safety and sustainability. Reinforcement Learning (RL) has emerged as a promising approach to enhancin…
ManagementReinforcement Learning (RL)Traffic Signal ControlDynamic Context Adaptation and Information Flow Control in Transformers: Introducing the Evaluator Adjuster Unit and Gated Residual Connections
Transformers have revolutionized various domains of artificial intelligence due to their unique ability to model long-range dependencies in data. However, they lack in nuanced, context-dependent modulation of features an…
Systematicity Emerges in Transformers when Abstract Grammatical Roles Guide Attention
Systematicity is thought to be a key inductive bias possessed by humans that is lacking in standard natural language processing systems such as those utilizing transformers. In this work, we investigate the extent to whi…
Inductive BiasSystematic Generalization