Compositional De-Attention Networks
Attentional models are distinctly characterized by their ability to learn relative importance, i.e., assigning a different weight to input values. This paper proposes a new quasi-attention that is compositional in nature, i.e., learning whether to \textit{add}, \textit{subtract} or \textit{nullify} a certain vector when learning representations. This is strongly contrasted with vanilla attention, which simply re-weights input tokens. Our proposed \textit{Compositional De-Attention} (CoDA) is fundamentally built upon the intuition of both similarity and dissimilarity (negative affinity) when computing affinity scores, benefiting from a greater extent of expressiveness. We evaluate CoDA on six NLP tasks, i.e. open domain question answering, retrieval/ranking, natural language inference, machine translation, sentiment analysis and text2code generation. We obtain promising experimental results, achieving state-of-the-art performance on several tasks/datasets.
Code (0)
등록된 구현이 없습니다.
Tasks
Machine TranslationNatural Language InferenceOpen-Domain Question AnsweringQuestion AnsweringRetrievalSentiment AnalysisTranslationSimilar Papers 제목 키워드 기반
MoCA: Mixture-of-Components Attention for Scalable Compositional 3D Generation
Compositionality is critical for 3D object and scene generation, but existing part-aware 3D generation methods suffer from poor scalability due to quadratic global attention costs when increasing the number of components…
Scene Generation3D GenerationImproving Compositional Generalization in Semantic Parsing
Generalization of models to out-of-distribution (OOD) data has captured tremendous attention recently. Specifically, compositional generalization, i.e., whether a model generalizes to new structures built of components o…
DecoderSemantic ParsingCompositional Generalization for Neural Semantic Parsing via Span-level Supervised Attention
We describe a span-level supervised attention loss that improves compositional generalization in semantic parsers. Our approach builds on existing losses that encourage attention maps in neural sequence-to-sequence model…
Machine TranslationSemantic ParsingTranslationWord AlignmentLearning compositionally through attentive guidance
While neural network models have been successfully applied to domains that require substantial generalisation skills, recent studies have implied that they struggle when solving the task they are trained on requires infe…
Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis
Large-scale diffusion models have achieved state-of-the-art results on text-to-image synthesis (T2I) tasks. Despite their ability to generate high-quality yet creative images, we observe that attribution-binding and comp…
AttributeImage Generation