Fixed Encoder Self-Attention Patterns in Transformer-Based Machine Translation
Transformer-based models have brought a radical change to neural machine translation. A key feature of the Transformer architecture is the so-called multi-head attention mechanism, which allows the model to focus simultaneously on different parts of the input. However, recent works have shown that most attention heads learn simple, and often redundant, positional patterns. In this paper, we propose to replace all but one attention head of each encoder layer with simple fixed -- non-learnable -- attentive patterns that are solely based on position and do not require any external knowledge. Our experiments with different data sizes and multiple language pairs show that fixing the attention heads on the encoder side of the Transformer at training time does not impact the translation quality and even increases BLEU scores by up to 3 points in low-resource scenarios.
Code (0)
등록된 구현이 없습니다.
Tasks
Machine TranslationPositionTranslationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
ViTCoD: Vision Transformer Acceleration via Dedicated Algorithm and Accelerator Co-Design
Vision Transformers (ViTs) have achieved state-of-the-art performance on various vision tasks. However, ViTs' self-attention module is still arguably a major bottleneck, limiting their achievable hardware efficiency. Mea…
Data-Informed Global Sparseness in Attention Mechanisms for Deep Neural Networks
Attention mechanisms play a crucial role in the neural revolution of Natural Language Processing (NLP). With the growth of attention-based models, several pruning techniques have been developed to identify and exploit sp…
GPULanguage ModelingLanguage ModellingMachine Translation+1Localized Gaussians as Self-Attention Weights for Point Clouds Correspondence
Current data-driven methodologies for point cloud matching demand extensive training time and computational resources, presenting significant challenges for model deployment and application. In the point cloud matching t…
Hard-Coded Gaussian Attention for Neural Machine Translation
Recent work has questioned the importance of the Transformer's multi-headed attention for achieving high translation quality. We push further in this direction by developing a "hard-coded" attention variant without any l…
DecoderMachine TranslationTranslationAttention Enables Zero Approximation Error
Deep learning models have been widely applied in various aspects of daily life. Many variant models based on deep learning structures have achieved even better performances. Attention-based architectures have become almo…
Deep Learningimage-classificationImage Classification