Generating Diverse Translation by Manipulating Multi-Head Attention
Transformer model has been widely used on machine translation tasks and obtained state-of-the-art results. In this paper, we report an interesting phenomenon in its encoder-decoder multi-head attention: different attention heads of the final decoder layer align to different word translation candidates. We empirically verify this discovery and propose a method to generate diverse translations by manipulating heads. Furthermore, we make use of these diverse translations with the back-translation technique for better data augmentation. Experiment results show that our method generates diverse translations without severe drop in translation quality. Experiments also show that back-translation with these diverse translations could bring significant improvement on performance on translation tasks. An auxiliary experiment of conversation response generation task proves the effect of diversity as well.
Code (0)
등록된 구현이 없습니다.
Tasks
Data AugmentationDecoderDiversityMachine TranslationResponse GenerationTranslationWord TranslationSimilar Papers 제목 키워드 기반
Discrete Structural Planning for Generating Diverse Translations
Planning is important for humans when producing complex languages, which is a missing part in current language generation models. In this work, we add a planning phase in neural machine translation to control the global …
DiversityMachine TranslationSentenceText Generation+1SANVis: Visual Analytics for Understanding Self-Attention Networks
Attention networks, a deep neural network architecture inspired by humans' attention mechanism, have seen significant success in image captioning, machine translation, and many other applications. Recently, they have bee…
Image CaptioningMachine TranslationSentenceTranslationGaussian3Diff: 3D Gaussian Diffusion for 3D Full Head Synthesis and Editing
We present a novel framework for generating photorealistic 3D human head and subsequently manipulating and reposing them with remarkable flexibility. The proposed approach leverages an implicit function representation of…
Face ModelDiscrete Structural Planning for Neural Machine Translation
Structural planning is important for producing long sentences, which is a missing part in current language generation models. In this work, we add a planning phase in neural machine translation to control the coarse stru…
Machine TranslationSentenceText GenerationTranslationDiscrete Structural Planning for Neural Machine Translation
Structural planning is important for producing long sentences, which is a missing part in current language generation models. In this work, we add a planning phase in neural machine translation to control the coarse stru…
Machine TranslationSentenceText GenerationTranslation