The Case for Translation-Invariant Self-Attention in Transformer-Based Language Models
Mechanisms for encoding positional information are central for transformer-based language models. In this paper, we analyze the position embeddings of existing language models, finding strong evidence of translation invariance, both for the embeddings themselves and for their effect on self-attention. The degree of translation invariance increases during training and correlates positively with model performance. Our findings lead us to propose translation-invariant self-attention (TISA), which accounts for the relative position between tokens in an interpretable fashion without needing conventional position embeddings. Our proposal has several theoretical advantages over existing position-representation approaches. Experiments show that it improves on regular ALBERT on GLUE tasks, while only adding orders of magnitude less positional parameters.
Code (3)
Tasks
PositionTranslationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
SiT: Symmetry-Invariant Transformers for Generalisation in Reinforcement Learning
An open challenge in reinforcement learning (RL) is the effective deployment of a trained policy to new or slightly different situations as well as semantically-similar environments. We introduce Symmetry-Invariant Trans…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Alias-Free ViT: Fractional Shift Invariance via Linear Attention
Transformers have emerged as a competitive alternative to convnets in vision tasks, yet they lack the architectural inductive bias of convnets, which may hinder their potential performance. Specifically, Vision Transform…
Image ClassificationSE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks
We introduce the SE(3)-Transformer, a variant of the self-attention module for 3D point clouds and graphs, which is equivariant under continuous 3D roto-translations. Equivariance is important to ensure stable and predic…
TranslationSynchronous Syntactic Attention for Transformer Neural Machine Translation
This paper proposes a novel attention mechanism for Transformer Neural Machine Translation, {``}Synchronous Syntactic Attention,{''} inspired by synchronous dependency grammars. The mechanism synchronizes source-side and…
DecoderMachine TranslationTranslationDependency-Based Self-Attention for Transformer NMT
In this paper, we propose a new Transformer neural machine translation (NMT) model that incorporates dependency relations into self-attention on both source and target sides, dependency-based self-attention. The dependen…
DecoderMachine TranslationNMTSemantic Role Labeling+1