Self-Attention with Structural Position Representations
Although self-attention networks (SANs) have advanced the state-of-the-art on various NLP tasks, one criticism of SANs is their ability of encoding positions of input words (Shaw et al., 2018). In this work, we propose to augment SANs with structural position representations to model the latent structure of the input sentence, which is complementary to the standard sequential positional representations. Specifically, we use dependency tree to represent the grammatical structure of a sentence, and propose two strategies to encode the positional relationships among words in the dependency tree. Experimental results on NIST Chinese-to-English and WMT14 English-to-German translation tasks show that the proposed approach consistently boosts performance over both the absolute and relative sequential position representations.
Code (0)
등록된 구현이 없습니다.
Tasks
PositionSentenceTranslationSimilar Papers 제목 키워드 기반
Does Structural Attention Improve Compositional Representations in Vision-Language Models?
Although scaling self-supervised approaches has gained widespread success in Vision-Language pre-training, a number of works providing structural knowledge of visually-grounded semantics have recently shown incremental…
Visual ReasoningDiscourse Self-Attention for Discourse Element Identification in Argumentative Student Essays
This paper proposes to adapt self-attention to discourse level for modeling discourse elements in argumentative student essays. Specifically, we focus on two issues. First, we propose structural sentence positional encod…
SentenceResidual Dominance as a Structural Account of Last-Item Reliance in Causal Self-Attention Recommenders
Transformer-based sequential recommenders with causal self-attention often rely heavily on the most recent interaction at inference time, but how this behavior is structurally expressed in the representation used for pre…
Structure-Aware Transformer for Graph Representation Learning
The Transformer architecture has gained growing attention in graph representation learning recently, as it naturally overcomes several limitations of graph neural networks (GNNs) by avoiding their strict structural induc…
Emotion Recognition in ConversationGraph Representation LearningRepresentation LearningComposition, Attention, or Both?
In this paper, we propose a novel architecture called Composition Attention Grammars (CAGs) that recursively compose subtrees into a single vector representation with a composition function, and selectively attend to pre…