Disentangling Neuron Representations with Concept Vectors
Mechanistic interpretability aims to understand how models store representations by breaking down neural networks into interpretable units. However, the occurrence of polysemantic neurons, or neurons that respond to multiple unrelated features, makes interpreting individual neurons challenging. This has led to the search for meaningful vectors, known as concept vectors, in activation space instead of individual neurons. The main contribution of this paper is a method to disentangle polysemantic neurons into concept vectors encapsulating distinct features. Our method can search for fine-grained concepts according to the user's desired level of concept separation. The analysis shows that polysemantic neurons can be disentangled into directions consisting of linear combinations of neurons. Our evaluations show that the concept vectors found encode coherent, human-understandable features.
Code (1)
Similar Papers 제목 키워드 기반
RAVEL: Evaluating Interpretability Methods on Disentangling Language Model Representations
Individual neurons participate in the representation of multiple high-level concepts. To what extent can different interpretability methods successfully disentangle these roles? To help address this question, we introduc…
AttributeLanguage ModelingLanguage ModellingGeometric operations implemented by conformal geometric algebra neural nodes
Geometric algebra is an optimal frame work for calculating with vectors. The geometric algebra of a space includes elements that represent all the its subspaces (lines, planes, volumes, ...). Conformal geometric algebra …
Disentangling MLP Neuron Weights in Vocabulary Space
Interpreting the information encoded in language model weights remains a fundamental challenge in mechanistic interpretability. In this work, we introduce ROTATE (Rotation-Optimized Token Alignment in weighT spacE), a da…
A Disentangling Invertible Interpretation Network for Explaining Latent Representations
Neural networks have greatly boosted performance in computer vision by learning powerful representations of input data. The drawback of end-to-end training for maximal overall performance are black-box models whose hidde…
Image GenerationImage ManipulationConcept Boundary Vectors
Machine learning models are trained with relatively simple objectives, such as next token prediction. However, on deployment, they appear to capture a more fundamental representation of their input data. It is of interes…