Towards Equipping Transformer with the Ability of Systematic Compositionality
One of the key factors in language productivity and human cognition is the ability of systematic compositionality, which refers to understanding composed unseen examples of seen primitives. However, recent evidence reveals that the Transformers have difficulty generalizing the composed context based on the seen primitives. To this end, we take the first step to propose a compositionality-aware Transformer called CAT and two novel pre-training tasks to facilitate systematic compositionality. We tentatively provide a successful implementation of a multi-layer CAT on the basis of the especially popular BERT. The experimental results demonstrate that CAT outperforms baselines on compositionality-aware tasks with minimal impact on the effectiveness on standardized language understanding tasks.
Code (1)
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Inducing Transformer’s Compositional Generalization Ability via Auxiliary Sequence Prediction Tasks
Systematic compositionality is an essential mechanism in human language, allowing the recombination of known parts to create novel expressions. However, existing neural models have been shown to lack this basic ability i…
Inducing Transformer's Compositional Generalization Ability via Auxiliary Sequence Prediction Tasks
Systematic compositionality is an essential mechanism in human language, allowing the recombination of known parts to create novel expressions. However, existing neural models have been shown to lack this basic ability i…
Enabling Systematic Generalization in Abstract Spatial Reasoning through Meta-Learning for Compositionality
Systematic generalization refers to the capacity to understand and generate novel combinations from known components. Despite recent progress by large language models (LLMs) across various domains, these models often fai…
Meta-LearningSpatial ReasoningSystematic GeneralizationTranslationInterpreting token compositionality in LLMs: A robustness analysis
Understanding the internal mechanisms of large language models (LLMs) is integral to enhancing their reliability, interpretability, and inference processes. We present Constituent-Aware Pooling (CAP), a methodology desig…
Definition ModellingComparing the Decision-Making Mechanisms by Transformers and CNNs via Explanation Methods
In order to gain insights about the decision-making of different visual recognition backbones, we propose two methodologies, sub-explanation counting and cross-testing, that systematically applies deep explanation algori…
Decision Making