Exploring the Impact of Layer Normalization for Zero-shot Neural Machine Translation

2023-05-16 · Zhuoyuan Mao, Raj Dabre, Qianying Liu, Haiyue Song, Chenhui Chu, Sadao Kurohashi

This paper studies the impact of layer normalization (LayerNorm) on zero-shot translation (ZST). Recent efforts for ZST often utilize the Transformer architecture as the backbone, with LayerNorm at the input of layers (PreNorm) set as the default. However, Xu et al. (2019) has revealed that PreNorm carries the risk of overfitting the training data. Based on this, we hypothesize that PreNorm may overfit supervised directions and thus have low generalizability for ZST. Through experiments on OPUS, IWSLT, and Europarl datasets for 54 ZST directions, we demonstrate that the original Transformer setting of LayerNorm after residual connections (PostNorm) consistently outperforms PreNorm by up to 12.3 BLEU points. We then study the performance disparities by analyzing the differences in off-target rates and structural variations between PreNorm and PostNorm. This study highlights the need for careful consideration of the LayerNorm setting for ZST.

📄 PDF Abstract BibTeX arXiv:2305.09312

Code (0)

등록된 구현이 없습니다.

Tasks

Machine Translation

Methods 이 논문이 사용한 방법론

Multi-Head Attention 설명 없음

Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Position-Wise Feed-Forward Layer 설명 없음

Residual Connection 설명 없음

Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…

Adam 설명 없음

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Exploring Database Normalization Effects on SQL Generation

2025-10-02 · Ryosuke Kohita arxiv

Schema design, particularly normalization, is a critical yet often overlooked factor in natural language to SQL (NL2SQL) systems. Most prior research evaluates models on fixed schemas, overlooking the influence of design…

Type prediction

Infrared Domain Adaptation with Zero-Shot Quantization

2024-08-25 · Burak Sevsay, Erdem Akagündüz

Quantization is one of the most popular techniques for reducing computation time and shrinking model size. However, ensuring the accuracy of quantized models typically involves calibration using training data, which may …

Domain Adaptation object-detection Object Detection Quantization

SeeDNorm: Self-Rescaled Dynamic Normalization

2025-10-26 · Wenrui Cai, Defa Zhu, Qingjie Liu, Qiyang Min arxiv

Normalization layer constitutes an essential component in neural networks. In transformers, the predominantly used RMSNorm constrains vectors to a unit hypersphere, followed by dimension-wise rescaling through a learnabl…

Zero-Shot Anomaly Detection via Batch Normalization

2023-02-15 · NeurIPS 2023 11 · Aodong Li, Chen Qiu, Marius Kloft, Padhraic Smyth 외

Anomaly detection (AD) plays a crucial role in many safety-critical application domains. The challenge of adapting an anomaly detector to drift in the normal data distribution, especially when no training data is availab…

Anomaly Detection Unsupervised Anomaly Detection zero-shot anomaly detection Zero-shot Generalization

Is Less More? Exploring Token Condensation as Training-free Adaptation for CLIP

2024-10-16 · Zixin Wang, Dong Gong, Sen Wang, Zi Huang 외

Contrastive language-image pre-training (CLIP) has shown remarkable generalization ability in image classification. However, CLIP sometimes encounters performance drops on downstream datasets during zero-shot inference. …

image-classification Image Classification Test-time Adaptation