SemFace: Pre-training Encoder and Decoder with a Semantic Interface for Neural Machine Translation

2021-08-01 · ACL 2021 5 · Shuo Ren, Long Zhou, Shujie Liu, Furu Wei, Ming Zhou, Shuai Ma

While pre-training techniques are working very well in natural language processing, how to pre-train a decoder and effectively use it for neural machine translation (NMT) still remains a tricky issue. The main reason is that the cross-attention module between the encoder and decoder cannot be pre-trained, and the combined encoder-decoder model cannot work well in the fine-tuning stage because the inputs of the decoder cross-attention come from unknown encoder outputs. In this paper, we propose a better pre-training method for NMT by defining a semantic interface (SemFace) between the pre-trained encoder and the pre-trained decoder. Specifically, we propose two types of semantic interfaces, including CL-SemFace which regards cross-lingual embeddings as an interface, and VQ-SemFace which employs vector quantized embeddings to constrain the encoder outputs and decoder inputs in the same language-independent space. We conduct massive experiments on six supervised translation pairs and three unsupervised pairs. Experimental results demonstrate that our proposed SemFace can effectively connect the pre-trained encoder and decoder, and achieves significant improvement by 3.7 and 1.5 BLEU points on the two tasks respectively compared with previous pre-training-based NMT models.

📄 PDF Abstract BibTeX

Code (0)

등록된 구현이 없습니다.

Tasks

Decoder Machine TranslationNMTTranslation

Methods 이 논문이 사용한 방법론

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Concatenated Skip Connection A Concatenated Skip Connection is a type of skip connection that seeks to reuse features by concatenating them to new layers, allowing more information to be retained from…

SemFaceEdit: Semantic Face Editing on Generative Radiance Manifolds

2025-06-28 · Shashikant Verma, Shanmuganathan Raman

Despite multiple view consistency offered by 3D-aware GAN techniques, the resulting images often lack the capacity for localized editing. In response, generative radiance manifolds emerge as an efficient approach for con…

Disentanglement

SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing

2021-10-14 · ACL 2022 5 · Junyi Ao, Rui Wang, Long Zhou, Chengyi Wang 외

Motivated by the success of T5 (Text-To-Text Transfer Transformer) in pre-trained natural language processing models, we propose a unified-modal SpeechT5 framework that explores the encoder-decoder pre-training for self-…

Automatic Speech Recognition Automatic Speech Recognition (ASR)Decoder Quantization+7

Language on Demand, Knowledge at Core: Composing LLMs with Encoder-Decoder Translation Models for Extensible Multilinguality

2026-03-18 · Mengyu Bu, Yang Feng arxiv

Large language models (LLMs) exhibit strong general intelligence, yet their multilingual performance remains highly imbalanced. Although LLMs encode substantial cross-lingual knowledge in a unified semantic space, they o…

General Knowledge

Decoder Denoising Pretraining for Semantic Segmentation

2022-05-23 · Emmanuel Brempong Asiedu, Simon Kornblith, Ting Chen, Niki Parmar 외

Semantic segmentation labels are expensive and time consuming to acquire. Hence, pretraining is commonly used to improve the label-efficiency of segmentation models. Typically, the encoder of a segmentation model is pret…

Decoder Denoising Segmentation Semantic Segmentation

Learning Structural Similarity of User Interface Layouts using Graph Networks

2020-08-01 · ECCV 2020 8 · Dipu Manandhar, Dan Ruta, John Collomosse

We propose a novel representation learning technique for measuring the similarity of user interface designs. A triplet network is used to learn a search embedding for layout similarity, with a hybrid encoder-decoder back…

Decoder Representation Learning Triplet