Dependency-Based Relative Positional Encoding for Transformer NMT
This paper proposes a new Transformer neural machine translation model that incorporates syntactic distances between two source words into the relative position representations of the self-attention mechanism. In particular, the proposed model encodes pair-wise relative depths on a source dependency tree, which are differences between the depths of the two source words, in the encoder{'}s self-attention. The experiments show that our proposed model achieves 0.5 point gain in BLEU on the Asian Scientific Paper Excerpt Corpus Japanese-to-English translation task.
Code (0)
등록된 구현이 없습니다.
Tasks
Machine TranslationNMTPositionTranslationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
RoFormer: Enhanced Transformer with Rotary Position Embedding
Position encoding recently has shown effective in the transformer architecture. It enables valuable supervision for dependency modeling between elements at different positions of the sequence. In this paper, we first inv…
PositionSemantic Text MatchingText ClassificationHoPE: Hyperbolic Rotary Positional Encoding for Stable Long-Range Dependency Modeling in Large Language Models
Positional encoding mechanisms enable Transformers to model sequential structure and long-range dependencies in text. While absolute positional encodings struggle with extrapolation to longer sequences due to fixed posit…
Linearized Relative Positional Encoding
Relative positional encoding is widely used in vanilla and linear transformers to represent positional information. However, existing encoding methods of a vanilla transformer are not always directly applicable to a line…
image-classificationImage ClassificationLanguage ModelingLanguage Modelling+2Dependency Transformer Grammars: Integrating Dependency Structures into Transformer Language Models
Syntactic Transformer language models aim to achieve better generalization through simultaneously modeling syntax trees and sentences. While prior work has been focusing on adding constituency-based structures to Transfo…
ARCInductive BiasLanguage ModelingLanguage ModellingPoD: Positional Dependency-Based Word Embedding for Aspect Term Extraction
Dependency context-based word embedding jointly learns the representations of word and dependency context, and has been proved effective in aspect term extraction. In this paper, we design the positional dependency-based…
Aspect Term Extraction and Sentiment ClassificationPOSPositionTerm Extraction