Recursive Tree Attention: Improving Semantic Representations with Syntactic Tree Structured Attention Mechanism
Attention mechanism has shown its effectiveness in state-of-the-art methods on various tasks in natural language processing (NLP). However, these methods are still using attention mechanism in plain, linear topological structures while the syntactic structures of natural languages are known to be hierarchical. Modeling in such syntactic structures like syntactic trees is proved to be superior in semantic representation learning. In this paper, to improve semantic representation learning by modelling attention mechanism in syntactic structures, we propose recursive tree attention, a syntactic tree structured attention mechanism which recursively calculate attention weights and summarize information through constituency parsing trees. The information of the children is cursively summarized to their parent through the constituency parsing tree. Then the summarized information from the root node is used as the semantic representations of the current sentence. Experimental results on the text classification tasks demonstrate the effectiveness of our proposed attention mechanism. The approaches with recursive tree attention outperform conventional attention mechanism and other syntactic tree based approaches.
Code (0)
등록된 구현이 없습니다.
Tasks
Constituency ParsingRepresentation LearningSentencetext-classificationText ClassificationSimilar Papers 제목 키워드 기반
Recursive Tree-Structured Self-Attention for Answer Sentence Selection
Syntactic structure is an important component of natural language text. Recent top-performing models in Answer Sentence Selection (AS2) use self-attention and transfer learning, but not syntactic structure. Tree structur…
Community Question AnsweringQuestion AnsweringSentenceTransfer LearningYou Only Need Attention to Traverse Trees
In recent NLP research, a topic of interest is universal sentence encoding, sentence representations that can be used in any supervised task. At the word sequence level, fully attention-based models suffer from two probl…
SentenceComposition, Attention, or Both?
In this paper, we propose a novel architecture called Composition Attention Grammars (CAGs) that recursively compose subtrees into a single vector representation with a composition function, and selectively attend to pre…
Structural Attention Neural Networks for improved sentiment analysis
We introduce a tree-structured attention neural network for sentences and small phrases and apply it to the problem of sentiment classification. Our model expands the current recursive models by incorporating structural …
General ClassificationSentiment AnalysisSentiment ClassificationA Re-ranking Model for Dependency Parser with Recursive Convolutional Neural Network
In this work, we address the problem to model all the nodes (words or phrases) in a dependency tree with the dense representations. We propose a recursive convolutional neural network (RCNN) architecture to capture synta…
Dependency ParsingRe-Ranking