InverseScope: Scalable Activation Inversion for Interpreting Large Language Models
Understanding the internal representations of large language models (LLMs) is a central challenge in interpretability research. Existing feature interpretability methods often rely on strong structural assumptions--such as linearity or sparsity--that may not hold in practice. In this work, we introduce InverseScope, an assumption-light and scalable framework for interpreting neural activations via input inversion. Given a target activation, InverseScope characterizes its encoded information by generating natural-language inputs that produce nearby activations, grounding abstract internal states in concrete language. To overcome the prohibitive cost of sampling in high-dimensional activation spaces, we propose a novel control-layer conditioning architecture that substantially improves sample efficiency compared to prior token-prepending approaches. We demonstrate that InverseScope reveals rich geometric structure in LLM representation spaces, including sentence-level linear analogies. The framework scales to state-of-the-art open-source models of up to 14B parameters and generalizes to out-of-distribution inputs, enabling systematic analysis of activation neighborhoods.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Building Better Activation Oracles
Activation Oracles (AOs) are promising methods for interpreting residual stream activations. However, current AOs face important issues, such as hallucinations and vagueness. Additionally, text-inversion confounds make t…
ARES: Scalable and Practical Gradient Inversion Attack in Federated Learning through Activation Recovery
Federated Learning (FL) enables collaborative model training by sharing model updates instead of raw data, aiming to protect user privacy. However, recent studies reveal that these shared updates can inadvertently leak s…
Federated LearningPredictive Concept Decoders: Training Scalable End-to-End Interpretability Assistants
Interpreting the internal activations of neural networks can produce more faithful explanations of their behavior, but is difficult due to the complex structure of activation space. Existing approaches to scalable interp…
N2G: A Scalable Approach for Quantifying Interpretable Neuron Representations in Large Language Models
Understanding the function of individual neurons within language models is essential for mechanistic interpretability research. We propose $\textbf{Neuron to Graph (N2G)}$, a tool which takes a neuron and its dataset exa…
Inference-Time Decomposition of Activations (ITDA): A Scalable Approach to Interpreting Large Language Models
Sparse autoencoders (SAEs) are a popular method for decomposing Large Langage Models (LLM) activations into interpretable latents. However, due to their substantial training cost, most academic research uses open-source …
GPULanguage ModelingLanguage Modelling