Structured Self-Attention Weights Encode Semantics in Sentiment Analysis
Neural attention, especially the self-attention made popular by the Transformer, has become the workhorse of state-of-the-art natural language processing (NLP) models. Very recent work suggests that the self-attention in the Transformer encodes syntactic information; Here, we show that self-attention scores encode semantics by considering sentiment analysis tasks. In contrast to gradient-based feature attribution methods, we propose a simple and effective Layer-wise Attention Tracing (LAT) method to analyze structured attention weights. We apply our method to Transformer models trained on two tasks that have surface dissimilarities, but share common semantics---sentiment analysis of movie reviews and time-series valence prediction in life story narratives. Across both tasks, words with high aggregated attention weights were rich in emotional semantics, as quantitatively validated by an emotion lexicon labeled by human annotators. Our results show that structured attention weights encode rich semantics in sentiment analysis, and match human interpretations of semantics.
Code (1)
Tasks
Sentiment AnalysisTime SeriesTime Series AnalysisMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Structured Self-AttentionWeights Encode Semantics in Sentiment Analysis
Neural attention, especially the self-attention made popular by the Transformer, has become the workhorse of state-of-the-art natural language processing (NLP) models. Very recent work suggests that the self-attention in…
Sentiment AnalysisTime SeriesTime Series AnalysisClass Semantics-based Attention for Action Detection
Action localization networks are often structured as a feature encoder sub-network and a localization sub-network, where the feature encoder learns to transform an input video to features that are useful for the localiza…
Action DetectionAction LocalizationHIRL: A General Framework for Hierarchical Image Representation Learning
Learning self-supervised image representations has been broadly studied to boost various visual understanding tasks. Existing methods typically learn a single level of image semantics like pairwise semantic similarity or…
Image ClusteringRepresentation LearningSelf-Supervised LearningSemantic Similarity+1An Analysis of Encoder Representations in Transformer-Based Machine Translation
The attention mechanism is a successful technique in modern NLP, especially in tasks like machine translation. The recently proposed network architecture of the \textit{Transformer} is based entirely on attention mechani…
Feature EngineeringMachine TranslationTransfer LearningTranslationDropKey
In this paper, we focus on analyzing and improving the dropout technique for self-attention layers of Vision Transformer, which is important while surprisingly ignored by prior works. In particular, we conduct researches…
Human-Object Interaction Detectionimage-classificationImage Classificationobject-detection+1