Multi-Head Attention with Disagreement Regularization
Multi-head attention is appealing for the ability to jointly attend to information from different representation subspaces at different positions. In this work, we introduce a disagreement regularization to explicitly encourage the diversity among multiple attention heads. Specifically, we propose three types of disagreement regularization, which respectively encourage the subspace, the attended positions, and the output representation associated with each attention head to be different from other heads. Experimental results on widely-used WMT14 English-German and WMT17 Chinese-English translation tasks demonstrate the effectiveness and universality of the proposed approach.
Code (0)
등록된 구현이 없습니다.
Tasks
DiversityTranslationSimilar Papers 제목 키워드 기반
Regularized Attentive Capsule Network for Overlapped Relation Extraction
Distantly supervised relation extraction has been widely applied in knowledge base construction due to its less requirement of human efforts. However, the automatically established training datasets in distant supervisio…
DiversityKnowledge Base ConstructionRelationRelation Extraction+1Orthogonality Constrained Multi-Head Attention For Keyword Spotting
Multi-head attention mechanism is capable of learning various representations from sequential data while paying attention to different subsequences, e.g., word-pieces or syllables in a spoken word. From the subsequences,…
Keyword SpottingAligning Cellular Sheaves with Classifier Attention for Interpretable Weakly-Supervised Pathology Localization
Weakly-supervised classification of whole-slide images with attention-based multiple instance learning (ABMIL) on top of foundation features now reaches near-saturation on Camelyon16 slide-level performance, but the corr…
Multiple Instance LearningScheduled DropHead: A Regularization Method for Transformer Models
In this paper, we introduce DropHead, a structured dropout method specifically designed for regularizing the multi-head attention mechanism, which is a key component of transformer, a state-of-the-art model for various N…
Machine Translationtext-classificationText ClassificationTranslationMulti-Head Attention Residuals
Transformers propagate information across depth through a single additive residual stream: every sublayer reads only the most recent state. Attention residuals relax this by letting each sublayer attend, through a learne…