Recurrent Attention Unit
Recurrent Neural Network (RNN) has been successfully applied in many sequence learning problems. Such as handwriting recognition, image description, natural language processing and video motion analysis. After years of development, researchers have improved the internal structure of the RNN and introduced many variants. Among others, Gated Recurrent Unit (GRU) is one of the most widely used RNN model. However, GRU lacks the capability of adaptively paying attention to certain regions or locations, so that it may cause information redundancy or loss during leaning. In this paper, we propose a RNN model, called Recurrent Attention Unit (RAU), which seamlessly integrates the attention mechanism into the interior of GRU by adding an attention gate. The attention gate can enhance GRU's ability to remember long-term memory and help memory cells quickly discard unimportant content. RAU is capable of extracting information from the sequential data by adaptively selecting a sequence of regions or locations and pay more attention to the selected regions during learning. Extensive experiments on image classification, sentiment classification and language modeling show that RAU consistently outperforms GRU and other baseline methods.
Code (0)
등록된 구현이 없습니다.
Tasks
General ClassificationHandwriting Recognitionimage-classificationImage ClassificationImage DescriptionLanguage ModelingLanguage ModellingSentiment AnalysisSentiment ClassificationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Channel Recurrent Attention Networks for Video Pedestrian Retrieval
Full attention, which generates an attention value per element of the input feature maps, has been successfully demonstrated to be beneficial in visual tasks. In this work, we propose a fully attentional network, termed …
Person RetrievalRetrievalEfficiently applying attention to sequential data with the Recurrent Discounted Attention unit
Recurrent Neural Networks architectures excel at processing sequences by modelling dependencies over different timescales. The recently introduced Recurrent Weighted Average (RWA) unit captures long term dependencies far…
Arabic Offensive Language Detection with Attention-based Deep Neural Networks
In this paper, we tackle the problem of offensive language and hate speech detection. We proposed our methods for data preprocessing and balancing, and then we presented our Convolutional Neural Network (CNN) and bidirec…
Abusive LanguageBIG-bench Machine LearningHate Speech DetectionDual Recurrent Attention Units for Visual Question Answering
Visual Question Answering (VQA) requires AI models to comprehend data in two domains, vision and text. Current state-of-the-art models use learned attention mechanisms to extract relevant information from the input domai…
Question AnsweringVisual Question AnsweringVisual Question Answering (VQA)Attention is all you need for Videos: Self-attention based Video Summarization using Universal Transformers
Video Captioning and Summarization have become very popular in the recent years due to advancements in Sequence Modelling, with the resurgence of Long-Short Term Memory networks (LSTMs) and introduction of Gated Recurren…
AllDense Video CaptioningDimensionality ReductionVideo Captioning+1