Using Perturbed Length-aware Positional Encoding for Non-autoregressive Neural Machine Translation

2021-07-29 · Yui Oka, Katsuhito Sudoh, Satoshi Nakamura

Non-autoregressive neural machine translation (NAT) usually employs sequence-level knowledge distillation using autoregressive neural machine translation (AT) as its teacher model. However, a NAT model often outputs shorter sentences than an AT model. In this work, we propose sequence-level knowledge distillation (SKD) using perturbed length-aware positional encoding and apply it to a student model, the Levenshtein Transformer. Our method outperformed a standard Levenshtein Transformer by 2.5 points in bilingual evaluation understudy (BLEU) at maximum in a WMT14 German to English translation. The NAT model output longer sentences than the baseline NAT models.

📄 PDF Abstract BibTeX arXiv:2107.13689

Code (0)

등록된 구현이 없습니다.

Tasks

Knowledge Distillation Machine Translation Translation

Methods 이 논문이 사용한 방법론

Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Knowledge Distillation A very simple way to improve the performance of almost any machine learning algorithm is to train many different models on the same data and then to average their predictions.…

Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…

Position-Wise Feed-Forward Layer 설명 없음

Adam 설명 없음

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…

Context-aware Biases for Length Extrapolation

2025-03-11 · Ali Veisi, Amir Mansourian

Transformers' ability to generalize to longer sequences than they have been trained on, known as length extrapolation, degrades as sequence length increases. Most of Relative Positional Encoding (RPE) methods address thi…

Decoder

Positional Encoding Field

2025-10-23 · Yunpeng Bai, Haoxiang Li, Qixing Huang arxiv

Diffusion Transformers (DiTs) have emerged as the dominant architecture for visual generation, powering state-of-the-art image and video models. By representing images as patch tokens with positional encodings (PEs), DiT…

Novel View Synthesis Image Editing

SAC-MIL: Spatial-Aware Correlated Multiple Instance Learning for Histopathology Whole Slide Image Classification

2025-09-04 · Yu Bai, Zitong Yu, Haowen Tian, Xijing Wang 외 arxiv

We propose Spatial-Aware Correlated Multiple Instance Learning (SAC-MIL) for performing WSI classification. SAC-MIL consists of a positional encoding module to encode position information and a SAC block to perform full …

Multiple Instance Learning Image Classification

Trie-Aware Transformers for Generative Recommendation

2026-02-25 · Zhenxiang Xu, Jiawei Chen, Sirui Chen, Yong He 외 arxiv

Generative recommendation (GR) aligns with advances in generative AI by casting next-item prediction as token-level generation rather than score-based ranking. Most GR methods adopt a two-stage pipeline: (i) \textit{item…

Incorporating Noisy Length Constraints into Transformer with Length-aware Positional Encodings

2020-12-01 · COLING 2020 8 · Yui Oka, Katsuki Chousa, Katsuhito Sudoh, Satoshi Nakamura

Neural Machine Translation often suffers from an under-translation problem due to its limited modeling of output sequence lengths. In this work, we propose a novel approach to training a Transformer model using length co…

Machine Translation Prediction Sentence Translation