Measuring Affinity between Attention-Head Weight Subspaces via the Projection Kernel
Understanding relationships between attention heads is essential for interpreting the internal structure of Transformers, yet existing metrics do not capture this structure well. We focus on the subspaces spanned by attention-head weight matrices and quantify head-to-head relationships using the Projection Kernel (PK), a principal-angle-based measure of subspace similarity. Experiments show that PK reproduces known head-to-head interactions on the IOI task more clearly than prior metrics such as the Composition Score. We further introduce a framework to quantify the informativeness of PK distributions by comparing them with a reference distribution derived from random orthogonal subspaces. As an application, we analyze a directed graph constructed from PK and show that, in GPT2-small, L4H7 acts as a hub by functioning as an identity head.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Affinity Graph Supervision for Visual Recognition
Affinity graphs are widely used in deep architectures, including graph convolutional neural networks and attention networks. Thus far, the literature has focused on abstracting features from such graphs, while the learni…
image-classificationImage ClassificationLearning Affinity from Attention: End-to-End Weakly-Supervised Semantic Segmentation with Transformers
Weakly-supervised semantic segmentation (WSSS) with image-level labels is an important and challenging task. Due to the high training efficiency, end-to-end solutions for WSSS have received increasing attention from the …
Semantic SegmentationWeakly supervised Semantic SegmentationWeakly-Supervised Semantic SegmentationMeasuring the sequence-affinity landscape of antibodies with massively parallel titration curves
Despite the central role that antibodies play in the adaptive immune system and in biotechnology, much remains unknown about the quantitative relationship between an antibody's amino acid sequence and its antigen binding…
HCAF-DTA: drug-target binding affinity prediction with cross-attention fused hypergraph neural networks
Accurate prediction of the binding affinity between drugs and target proteins is a core task in computer-aided drug design. Existing deep learning methods tend to ignore the information of internal sub-structural feature…
Drug DesignPredictionTree DecompositionCross-Dimension Affinity Distillation for 3D EM Neuron Segmentation
Accurate 3D neuron segmentation from electron microscopy (EM) volumes is crucial for neuroscience research. However the complex neuron morphology often leads to over-merge and over-segmentation results. Recent advanc…
SegmentationTransfer Learning