Understanding Attention for Text Classification
Attention has been proven successful in many natural language processing (NLP) tasks. Recently, many researchers started to investigate the interpretability of attention on NLP tasks. Many existing approaches focused on examining whether the local attention weights could reflect the importance of input representations. In this work, we present a study on understanding the internal mechanism of attention by looking into the gradient update process, checking its behavior when approaching a local minimum during training. We propose to analyze for each word token the following two quantities: its polarity score and its attention score, where the latter is a global assessment on the token{'}s significance. We discuss conditions under which the attention mechanism may become more (or less) interpretable, and show how the interplay between the two quantities can contribute towards model performance.
Code (0)
등록된 구현이 없습니다.
Tasks
ClassificationGeneral Classificationtext-classificationText ClassificationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
A Global-Local Attention Mechanism for Relation Classification
Relation classification, a crucial component of relation extraction, involves identifying connections between two entities. Previous studies have predominantly focused on integrating the attention mechanism into relation…
ClassificationRelationRelation ClassificationRelation ExtractionAttention Flows: Analyzing and Comparing Attention Mechanisms in Language Models
Advances in language modeling have led to the development of deep attention-based models that are performant across a wide variety of natural language processing (NLP) problems. These language models are typified by a pr…
Deep AttentionLanguage ModelingLanguage ModellingSentenceText Classification Based on Knowledge Graphs and Improved Attention Mechanism
To resolve the semantic ambiguity in texts, we propose a model, which innovatively combines a knowledge graph with an improved attention mechanism. An existing knowledge base is utilized to enrich the text with relevant …
DecoderKnowledge Graphstext-classificationText ClassificationEAML: Ensemble Self-Attention-based Mutual Learning Network for Document Image Classification
In the recent past, complex deep neural networks have received huge interest in various document understanding tasks such as document image classification and document retrieval. As many document types have a distinct vi…
document-image-classificationDocument Image Classificationimage-classification基于堆叠式注意力网络的复杂话语领域分类方法(Complex Utterance Domain Classification Using Stacked Attention Networks)
“话语领域分类(utterance domain classification UDC)是口语语言理解(spoken lan-guage understanding SLU)中语义分析的关键步骤。尽管带注意力机制的递归神经网络已经得到了广泛的应用,并将UDC的研究进展提高到了一个新的水平,但是对于复杂的话语,如长度较长的话语或带有逗号的复合句的话语,有效的UDC仍然是一个挑战。本文提出一种基于堆叠式注意力网络的话语领域分类方法SAN-D…
domain classification