Effective Attention Sheds Light On Interpretability

2021-05-18 · Findings (ACL) 2021 8 · Kaiser Sun, Ana Marasović

An attention matrix of a transformer self-attention sublayer can provably be decomposed into two components and only one of them (effective attention) contributes to the model output. This leads us to ask whether visualizing effective attention gives different conclusions than interpretation of standard attention. Using a subset of the GLUE tasks and BERT, we carry out an analysis to compare the two attention matrices, and show that their interpretations differ. Effective attention is less associated with the features related to the language modeling pretraining such as the separator token, and it has more potential to illustrate linguistic features captured by the model for solving the end-task. Given the found differences, we recommend using effective attention for studying a transformer's behavior since it is more pertinent to the model output by design.

📄 PDF Abstract BibTeX arXiv:2105.08855

Code (1)

KaiserWhoLearns/Effective-Attention-Interpretability 공식 구현 pytorch

Tasks

Language ModelingLanguage Modelling

Methods 이 논문이 사용한 방법론

Multi-Head Attention 설명 없음

Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Linear Warmup With Linear Decay Linear Warmup With Linear Decay is a learning rate schedule in which we increase the learning rate linearly for $n$ updates and then linearly decay afterwards.

Attention Dropout Attention Dropout is a type of dropout used in attention-based architectures, where elements are randomly dropped out of the…

WordPiece 설명 없음

BR-NPA: A Non-Parametric High-Resolution Attention Model to improve the Interpretability of Attention

2021-06-04 · Tristan Gomez, Suiyi Ling, Thomas Fréour, Harold Mouchère

The prevalence of employing attention mechanisms has brought along concerns on the interpretability of attention distributions. Although it provides insights about how a model is operating, utilizing attention as the exp…

Classification Fine-Grained Image Classification image-classification Image Classification+1

Towards Interpretable and Efficient Attention: Compressing All by Contracting a Few

2025-09-21 · Qishuai Wen, Zhiyuan Huang, Chun-Guang Li arxiv

Attention mechanisms have achieved significant empirical success in multiple fields, but their underlying optimization objectives remain unclear yet. Moreover, the quadratic complexity of self-attention has become increa…

Deep learning interpretability for rough volatility

2024-11-28 · Bo Yuan, Damiano Brigo, Antoine Jacquier, Nicola Pede

Deep learning methods have become a widespread toolbox for pricing and calibration of financial models. While they often provide new directions and research results, their `black box' nature also results in a lack of int…

Deep Learning

Clustering Text Using Attention

2022-01-08 · Lovedeep Singh

Clustering Text has been an important problem in the domain of Natural Language Processing. While there are techniques to cluster text based on using conventional clustering techniques on top of contextual or non-context…

Clustering

MINAR: Mechanistic Interpretability for Neural Algorithmic Reasoning

2026-02-24 · Jesse He, Helen Jenne, Max Vargas, Davis Brown 외 arxiv

The recent field of neural algorithmic reasoning (NAR) studies the ability of graph neural networks (GNNs) to emulate classical algorithms like Bellman-Ford, a phenomenon known as algorithmic alignment. At the same time,…