Neural Attention Models for Sequence Classification: Analysis and Application to Key Term Extraction and Dialogue Act Detection
Recurrent neural network architectures combining with attention mechanism, or neural attention model, have shown promising performance recently for the tasks including speech recognition, image caption generation, visual question answering and machine translation. In this paper, neural attention model is applied on two sequence classification tasks, dialogue act detection and key term extraction. In the sequence labeling tasks, the model input is a sequence, and the output is the label of the input sequence. The major difficulty of sequence labeling is that when the input sequence is long, it can include many noisy or irrelevant part. If the information in the whole sequence is treated equally, the noisy or irrelevant part may degrade the classification performance. The attention mechanism is helpful for sequence classification task because it is capable of highlighting important part among the entire sequence for the classification task. The experimental results show that with the attention mechanism, discernible improvements were achieved in the sequence labeling task considered here. The roles of the attention mechanism in the tasks are further analyzed and visualized in this paper.
Code (0)
등록된 구현이 없습니다.
Tasks
Caption GenerationClassificationGeneral ClassificationMachine TranslationQuestion Answeringspeech-recognitionSpeech RecognitionTerm ExtractionTranslationVisual Question AnsweringVisual Question Answering (VQA)Similar Papers 제목 키워드 기반
Temporal Attention-Gated Model for Robust Sequence Classification
Typical techniques for sequence classification are designed for well-segmented sequences which have been edited to remove noisy or irrelevant parts. Therefore, such methods cannot be easily applied on noisy sequences exp…
ClassificationGeneral ClassificationmodelSentiment AnalysisRandom Feature Attention
Transformers are state-of-the-art models for a variety of sequence modeling tasks. At their core is an attention function which models pairwise interactions between the inputs at every timestep. While attention is powerf…
Language ModelingLanguage ModellingMachine Translationtext-classification+2A Survey of Classical And Quantum Sequence Models
Our primary objective is to conduct a brief survey of various classical and quantum neural net sequence models, which includes self-attention and recurrent neural networks, with a focus on recent quantum approaches propo…
Classificationimage-classificationImage ClassificationSurveySemantic-Unit-Based Dilated Convolution for Multi-Label Text Classification
We propose a novel model for multi-label text classification, which is based on sequence-to-sequence learning. The model generates higher-level semantic unit representations with multi-level dilated convolution as well a…
ClassificationGeneral ClassificationMulti Label Text ClassificationMulti-Label Text Classification+2Big Bird: Transformers for Longer Sequences
Transformers-based models, such as BERT, have been one of the most successful deep learning models for NLP. Unfortunately, one of their core limitations is the quadratic dependency (mainly in terms of memory) on the sequ…
Linguistic AcceptabilityNatural Language InferenceQuestion AnsweringSemantic Textual Similarity+2