NAM: Normalization-based Attention Module
Recognizing less salient features is the key for model compression. However, it has not been investigated in the revolutionary attention mechanisms. In this work, we propose a novel normalization-based attention module (NAM), which suppresses less salient weights. It applies a weight sparsity penalty to the attention modules, thus, making them more computational efficient while retaining similar performance. A comparison with three other attention mechanisms on both Resnet and Mobilenet indicates that our method results in higher accuracy. Code for this paper can be publicly accessed at https://github.com/Christian-lyc/NAM.
Code (1)
Tasks
Model CompressionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
SAFIN: Arbitrary Style Transfer With Self-Attentive Factorized Instance Normalization
Artistic style transfer aims to transfer the style characteristics of one image onto another image while retaining its content. Existing approaches commonly leverage various normalization techniques, although these face …
Style TransferU-GAT-IT: Unsupervised Generative Attentional Networks with Adaptive Layer-Instance Normalization for Image-to-Image Translation
We propose a novel method for unsupervised image-to-image translation, which incorporates a new attention module and a new learnable normalization function in an end-to-end manner. The attention module guides our model t…
Fundus to Angiography GenerationImage-to-Image TranslationTranslationUnsupervised Image-To-Image TranslationAttention Normalization Impacts Cardinality Generalization in Slot Attention
Object-centric scene decompositions are important representations for downstream tasks in fields such as computer vision and robotics. The recently proposed Slot Attention module, already leveraged by several derivative …
Image SegmentationObjectObject TrackingSemantic Segmentation+1Normalized and Geometry-Aware Self-Attention Network for Image Captioning
Self-attention (SA) network has shown profound value in image captioning. In this paper, we improve SA from two aspects to promote the performance of image captioning. First, we propose Normalized Self-Attention (NSA), a…
Image CaptioningMachine TranslationQuestion AnsweringTranslation+3iRNN: Integer-only Recurrent Neural Network
Recurrent neural networks (RNN) are used in many real-world text and speech applications. They include complex modules such as recurrence, exponential-based activation, gate interaction, unfoldable normalization, bi-dire…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Quantizationspeech-recognition+1