Masked Attention as a Mechanism for Improving Interpretability of Vision Transformers
Vision Transformers are at the heart of the current surge of interest in foundation models for histopathology. They process images by breaking them into smaller patches following a regular grid, regardless of their content. Yet, not all parts of an image are equally relevant for its understanding. This is particularly true in computational pathology where background is completely non-informative and may introduce artefacts that could mislead predictions. To address this issue, we propose a novel method that explicitly masks background in Vision Transformers' attention mechanism. This ensures tokens corresponding to background patches do not contribute to the final image representation, thereby improving model robustness and interpretability. We validate our approach using prostate cancer grading from whole-slide images as a case study. Our results demonstrate that it achieves comparable performance with plain self-attention while providing more accurate and clinically meaningful attention heatmaps.
Code (0)
등록된 구현이 없습니다.
Tasks
whole slide imagesSimilar Papers 제목 키워드 기반
Attention Mechanism, Transformers, BERT, and GPT: Tutorial and Survey
This is a tutorial and survey paper on the attention mechanism, transformers, BERT, and GPT. We first explain attention mechanism, sequence-to-sequence model without and with attention, self-attention, and attention in d…
DecoderDeep AttentionNatural Language InferenceSurvey+1MaiT: integrating spatial locality into image transformers with attention masks
Though image transformers have shown competitive results with convolutional neural networks in computer vision tasks, lacking inductive biases such as locality still poses problems in terms of model efficiency especially…
Attention mechanisms in neural networks
Attention mechanisms represent a fundamental paradigm shift in neural network architectures, enabling models to selectively focus on relevant portions of input sequences through learned weighting functions. This monograp…
Representation LearningImage ClassificationInterpretable Vision Transformers in Image Classification via SVDA
Vision Transformers (ViTs) have achieved state-of-the-art performance in image classification, yet their attention mechanisms often remain opaque and exhibit dense, non-structured behaviors. In this work, we adapt our pr…
Image ClassificationModel CompressionMATIS: Masked-Attention Transformers for Surgical Instrument Segmentation
We propose Masked-Attention Transformers for Surgical Instrument Segmentation (MATIS), a two-stage, fully transformer-based method that leverages modern pixel-wise attention mechanisms for instrument segmentation. MATIS …
Segmentation