Systematic Generalization with Edge Transformers
Recent research suggests that systematic generalization in natural language understanding remains a challenge for state-of-the-art neural models such as Transformers and Graph Neural Networks. To tackle this challenge, we propose Edge Transformer, a new model that combines inspiration from Transformers and rule-based symbolic AI. The first key idea in Edge Transformers is to associate vector states with every edge, that is, with every pair of input nodes -- as opposed to just every node, as it is done in the Transformer model. The second major innovation is a triangular attention mechanism that updates edge representations in a way that is inspired by unification from logic programming. We evaluate Edge Transformer on compositional generalization benchmarks in relational reasoning, semantic parsing, and dependency parsing. In all three settings, the Edge Transformer outperforms Relation-aware, Universal and classical Transformer baselines.
Code (1)
Tasks
Dependency ParsingNatural Language UnderstandingRelational ReasoningSemantic ParsingSystematic GeneralizationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Loop, Think, & Generalize: Implicit Reasoning in Recurrent-Depth Transformers
We study implicit reasoning, i.e. the ability to combine knowledge or rules within a single forward pass. While transformer-based large language models store substantial factual knowledge and rules, they often fail to co…
Transformer Module Networks for Systematic Generalization in Visual Question Answering
Transformers achieve great performance on Visual Question Answering (VQA). However, their systematic generalization capabilities, i.e., handling novel combinations of known concepts, is unclear. We reveal that Neural Mod…
Question AnsweringSystematic GeneralizationVisual Question AnsweringVisual Question Answering (VQA)Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalization
We study whether transformers can learn to implicitly reason over parametric knowledge, a skill that even the most capable language models struggle with. Focusing on two representative reasoning types, composition and co…
Systematicity Emerges in Transformers when Abstract Grammatical Roles Guide Attention
Systematicity is thought to be a key inductive bias possessed by humans that is lacking in standard natural language processing systems such as those utilizing transformers. In this work, we investigate the extent to whi…
Inductive BiasSystematic GeneralizationMeasuring Systematic Generalization in Neural Proof Generation with Transformers
We are interested in understanding how well Transformer language models (TLMs) can perform reasoning tasks when trained on knowledge encoded in the form of natural language. We investigate their systematic generalization…
Automated Theorem ProvingLogical ReasoningSystematic Generalization