Making Transformers Solve Compositional Tasks
Several studies have reported the inability of Transformer models to generalize compositionally, a key type of generalization in many NLP tasks such as semantic parsing. In this paper we explore the design space of Transformer models showing that the inductive biases given to the model by several design decisions significantly impact compositional generalization. We identified Transformer configurations that generalize compositionally significantly better than previously reported in the literature in many compositional tasks. We achieve state-of-the-art results in a semantic parsing compositional generalization benchmark (COGS), and a string edit operation composition benchmark (PCFG).
Code (0)
등록된 구현이 없습니다.
Tasks
Semantic ParsingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Making Transformers Solve Compositional Tasks
Several studies have reported the inability of Transformer models to generalize compositionally, a key type of generalization in many NLP tasks such as semantic parsing. In this paper we explore the design space of Trans…
Semantic ParsingAttention as a Hypernetwork
Transformers can under some circumstances generalize to novel problem instances whose constituent parts might have been encountered during training but whose compositions have not. What mechanisms underlie this ability f…
Strassen Attention: Unlocking Compositional Abilities in Transformers Based on a New Lower Bound Method
We propose a novel method to evaluate the theoretical limits of Transformers, allowing us to prove the first lower bounds against one-layer softmax Transformers with infinite precision. We establish those bounds for thre…
AllDisentangling Reasoning Capabilities from Language Models with Compositional Reasoning Transformers
This paper presents ReasonFormer, a unified reasoning framework for mirroring the modular and compositional reasoning process of humans in complex decision-making. Inspired by dual-process theory in cognitive science, th…
Decision MakingIn-Context Compositional Learning via Sparse Coding Transformer
Transformer architectures have achieved remarkable success across language, vision, and multimodal tasks, and there is growing demand for them to address in-context compositional learning tasks. In these tasks, models so…