Integrating Dependency Tree Into Self-attention for Sentence Representation
Recent progress on parse tree encoder for sentence representation learning is notable. However, these works mainly encode tree structures recursively, which is not conducive to parallelization. On the other hand, these works rarely take into account the labels of arcs in dependency trees. To address both issues, we propose Dependency-Transformer, which applies a relation-attention mechanism that works in concert with the self-attention mechanism. This mechanism aims to encode the dependency and the spatial positional relations between nodes in the dependency tree of sentences. By a score-based method, we successfully inject the syntax information without affecting Transformer's parallelizability. Our model outperforms or is comparable to the state-of-the-art methods on four tasks for sentence representation and has obvious advantages in computational efficiency.
Code (0)
등록된 구현이 없습니다.
Tasks
Computational EfficiencyRepresentation LearningSentenceSimilar Papers 제목 키워드 기반
Dependency Transformer Grammars: Integrating Dependency Structures into Transformer Language Models
Syntactic Transformer language models aim to achieve better generalization through simultaneously modeling syntax trees and sentences. While prior work has been focusing on adding constituency-based structures to Transfo…
ARCInductive BiasLanguage ModelingLanguage ModellingVisual Dependency Transformers: Dependency Tree Emerges from Reversed Attention
Humans possess a versatile mechanism for extracting structured representations of our visual world. When looking at an image, we can decompose the scene into entities and their parts as well as obtain the dependencies be…
Joint Learning of Local and Global Features for Aspect-based Sentiment Classification
Aspect-based sentiment classification (ASC) aims to judge the sentiment polarity conveyed by the given aspect term in a sentence. The sentiment polarity is not only determined by the local context but also related to the…
Graph AttentionRepresentation LearningSentenceSentiment Analysis+2Dependency-Based Relative Positional Encoding for Transformer NMT
This paper proposes a new Transformer neural machine translation model that incorporates syntactic distances between two source words into the relative position representations of the self-attention mechanism. In particu…
Machine TranslationNMTPositionTranslationDependency Position Encoding for Relation Extraction
Leveraging the dependency tree of the input sentence is able to improve the model performance for relation extraction. A challenging issue is how to remove confusions from the tree. Efforts have been made to utilize the …
PositionRelationRelation ExtractionSentence