Neural Machine Translation with Joint Representation

2020-02-16 · Yanyang Li, Qiang Wang, Tong Xiao, Tongran Liu, Jingbo Zhu

Though early successes of Statistical Machine Translation (SMT) systems are attributed in part to the explicit modelling of the interaction between any two source and target units, e.g., alignment, the recent Neural Machine Translation (NMT) systems resort to the attention which partially encodes the interaction for efficiency. In this paper, we employ Joint Representation that fully accounts for each possible interaction. We sidestep the inefficiency issue by refining representations with the proposed efficient attention operation. The resulting Reformer models offer a new Sequence-to- Sequence modelling paradigm besides the Encoder-Decoder framework and outperform the Transformer baseline in either the small scale IWSLT14 German-English, English-German and IWSLT15 Vietnamese-English or the large scale NIST12 Chinese-English translation tasks by about 1 BLEU point.We also propose a systematic model scaling approach, allowing the Reformer model to beat the state-of-the-art Transformer in IWSLT14 German-English and NIST12 Chinese-English with about 50% fewer parameters. The code is publicly available at https://github.com/lyy1994/reformer.

📄 PDF Abstract BibTeX arXiv:2002.06546

Code (1)

lyy1994/reformer 공식 구현 pytorch

Tasks

Decoder Machine TranslationNMTTranslation

Methods 이 논문이 사용한 방법론

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…

Position-Wise Feed-Forward Layer 설명 없음

LSH Attention 설명 없음

Adafactor Adafactor is a stochastic optimization method based on Adam that reduces memory usage while retaining the empirical benefits of…

Reversible Residual Block 설명 없음

Residual Connection 설명 없음

SentencePiece 설명 없음

Auto-Encoding Variational Neural Machine Translation

2018-07-27 · WS 2019 8 · Bryan Eikema, Wilker Aziz

We present a deep generative model of bilingual sentence pairs for machine translation. The model generates source and target sentences jointly from a shared latent representation and is parameterised by neural networks.…

Machine Translation Sentence Translation Variational Inference

Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities

2018-12-19 · Hai Pham, Paul Pu Liang, Thomas Manzini, Louis-Philippe Morency 외

Multimodal sentiment analysis is a core research area that studies speaker sentiment expressed from the language, visual, and acoustic modalities. The central challenge in multimodal learning involves inferring joint rep…

Machine Translation Multimodal Sentiment Analysis Sentiment Analysis Translation

Learning Contextualized Sentence Representations for Document-Level Neural Machine Translation

2020-03-30 · Pei Zhang, Xu Zhang, Wei Chen, Jian Yu 외

Document-level machine translation incorporates inter-sentential dependencies into the translation of a source sentence. In this paper, we propose a new framework to model cross-sentence dependencies by training neural m…

Document Level Machine Translation Machine Translation NMT Sentence+2

Learning Joint Multilingual Sentence Representations with Neural Machine Translation

2017-04-13 · WS 2017 8 · Holger Schwenk, Matthijs Douze

In this paper, we use the framework of neural machine translation to learn joint sentence representations across six very different languages. Our aim is that a representation which is independent of the language, is lik…

Joint Multilingual Sentence Representations Machine Translation Sentence Translation

BERT-JAM: Boosting BERT-Enhanced Neural Machine Translation with Joint Attention

2020-11-09 · Zhebin Zhang, Sai Wu, Dawei Jiang, Gang Chen

BERT-enhanced neural machine translation (NMT) aims at leveraging BERT-encoded representations for translation tasks. A recently proposed approach uses attention mechanisms to fuse Transformer's encoder and decoder layer…

Decoder Machine Translation NMT Translation