Decoding Layer Saliency in Language Transformers
In this paper, we introduce a strategy for identifying textual saliency in large-scale language models applied to classification tasks. In visual networks where saliency is more well-studied, saliency is naturally localized through the convolutional layers of the network; however, the same is not true in modern transformer-stack networks used to process natural language. We adapt gradient-based saliency methods for these networks, propose a method for evaluating the degree of semantic coherence of each layer, and demonstrate consistent improvement over numerous other methods for textual saliency on multiple benchmark classification datasets. Our approach requires no additional training or access to labelled data, and is comparatively very computationally efficient.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Transformer-based Video Saliency Prediction with High Temporal Dimension Decoding
In recent years, finding an effective and efficient strategy for exploiting spatial and temporal information has been a hot research topic in video saliency prediction (VSP). With the emergence of spatio-temporal transfo…
DecoderSaliency PredictionVideo Saliency PredictionLearning Saliency From Fixations
We present a novel approach for saliency prediction in images, leveraging parallel decoding in transformers to learn saliency solely from fixation maps. Models typically rely on continuous saliency maps, to overcome the …
DecoderPredictionSaliency PredictionFull-bandwidth transformer
Autoregressive transformers compute along two axes: horizontally across generated tokens, and vertically through model depth. Dense attention gives each token broad horizontal access to the past, but the vertical feedbac…
T^2MLR: Transformer with Temporal Middle-Layer Recurrence
Transformer reasoning is limited by autoregressive decoding, which repeat edly compresses rich hidden computation through token space and makes it difficult for intermediate reasoning states to persist across time. We in…
Balancing Cost and Benefit with Tied-Multi Transformers
We propose and evaluate a novel procedure for training multiple Transformers with tied parameters which compresses multiple models into one enabling the dynamic choice of the number of encoder and decoder layers during d…
DecoderKnowledge DistillationMachine TranslationModel Compression+1