Encoding Sentence Position in Context-Aware Neural Machine Translation with Concatenation

2023-02-13 · Lorenzo Lupo, Marco Dinarelli, Laurent Besacier

Context-aware translation can be achieved by processing a concatenation of consecutive sentences with the standard Transformer architecture. This paper investigates the intuitive idea of providing the model with explicit information about the position of the sentences contained in the concatenation window. We compare various methods to encode sentence positions into token representations, including novel methods. Our results show that the Transformer benefits from certain sentence position encoding methods on English to Russian translation if trained with a context-discounted loss (Lupo et al., 2022). However, the same benefits are not observed in English to German. Further empirical efforts are necessary to define the conditions under which the proposed approach is beneficial.

📄 PDF Abstract BibTeX arXiv:2302.06459

Code (1)

lorelupo/focused-concat 공식 구현 pytorch

Tasks

Machine TranslationPositionSentenceTranslation

Methods 이 논문이 사용한 방법론

Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Multi-Head Attention 설명 없음

Position-Wise Feed-Forward Layer 설명 없음

Position-aware Self-attention with Relative Positional Encodings for Slot Filling

2018-07-09 · Ivan Bilan, Benjamin Roth

This paper describes how to apply self-attention with relative positional encodings to the task of relation extraction. We propose to use the self-attention encoder layer together with an additional position-aware attent…

Position Relation Extraction Sentence slot-filling+1

Using Perturbed Length-aware Positional Encoding for Non-autoregressive Neural Machine Translation

2021-07-29 · Yui Oka, Katsuhito Sudoh, Satoshi Nakamura

Non-autoregressive neural machine translation (NAT) usually employs sequence-level knowledge distillation using autoregressive neural machine translation (AT) as its teacher model. However, a NAT model often outputs shor…

Knowledge Distillation Machine Translation Translation

Hierarchical Attention Based Position-Aware Network for Aspect-Level Sentiment Analysis

2018-10-01 · CONLL 2018 10 · Lishuang Li, Yang Liu, AnQiao Zhou

Aspect-level sentiment analysis aims to identify the sentiment of a specific target in its context. Previous works have proved that the interactions between aspects and the contexts are important. On this basis, we also …

Aspect-Based Sentiment Analysis (ABSA)Feature Engineering Position Sentence+1

Context- and Sequence-Aware Convolutional Recurrent Encoder for Neural Machine Translation

2021-01-11 · Ritam Mallick, Seba Susan, Vaibhaw Agrawal, Rizul Garg 외

Neural Machine Translation model is a sequence-to-sequence converter based on neural networks. Existing models use recurrent neural networks to construct both the encoder and decoder modules. In alternative research, the…

Decoder Machine Translation Sentence Translation

Segatron: Segment-Aware Transformer for Language Modeling and Understanding

2020-04-30 · He Bai, Peng Shi, Jimmy Lin, Yuqing Xie 외

Transformers are powerful for sequence modeling. Nearly all state-of-the-art language models and pre-trained language models are based on the Transformer architecture. However, it distinguishes sequential tokens only wit…

Language Modeling Language Modelling Masked Language Modeling Position+2