Scaling sparse feature circuit finding for in-context learning

2025-04-18 · Dmitrii Kharlapenko, Stepan Shabalin, Fazl Barez, Arthur Conmy, Neel Nanda

Sparse autoencoders (SAEs) are a popular tool for interpreting large language model activations, but their utility in addressing open questions in interpretability remains unclear. In this work, we demonstrate their effectiveness by using SAEs to deepen our understanding of the mechanism behind in-context learning (ICL). We identify abstract SAE features that (i) encode the model's knowledge of which task to execute and (ii) whose latent vectors causally induce the task zero-shot. This aligns with prior work showing that ICL is mediated by task vectors. We further demonstrate that these task vectors are well approximated by a sparse sum of SAE latents, including these task-execution features. To explore the ICL mechanism, we adapt the sparse feature circuits methodology of Marks et al. (2024) to work for the much larger Gemma-1 2B model, with 30 times as many parameters, and to the more complex task of ICL. Through circuit finding, we discover task-detecting features with corresponding SAE latents that activate earlier in the prompt, that detect when tasks have been performed. They are causally linked with task-execution features through the attention and MLP sublayers.

📄 PDF Abstract BibTeX arXiv:2504.13756

Code (0)

등록된 구현이 없습니다.

Tasks

In-Context LearningLarge Language Model

Methods 이 논문이 사용한 방법론

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Attention 설명 없음

Weight-sparse transformers have interpretable circuits

2025-11-17 · Leo Gao, Achyuta Rajaram, Jacob Coxon, Soham V. Govande 외 arxiv

Finding human-understandable circuits in language models is a central goal of the field of mechanistic interpretability. We train models to have more understandable circuits by constraining most of their weights to be ze…

Sparse Visual Thought Circuits in Vision-Language Models

2026-03-26 · Yunpeng Zhou arxiv

Sparse autoencoders (SAEs) improve interpretability in multimodal models, but it remains unclear whether SAE features form modular, composable units for reasoning-an assumption underlying many intervention-based steering…

A Recurrent Neural Circuit Mechanism of Temporal-scaling Equivariant Representation

2023-09-21 · NeurIPS 2023 11

Time perception is critical in our daily life. An important feature of time perception is temporal scaling (TS): the ability to generate temporal sequences (e.g., motor actions) at different speeds. However, it is largel…

Towards Understanding the Nature of Attention with Low-Rank Sparse Decomposition

2025-04-29 · Zhengfu He, Junxuan Wang, Rui Lin, Xuyang Ge 외

We propose Low-Rank Sparse Attention (Lorsa), a sparse replacement model of Transformer attention layers to disentangle original Multi Head Self Attention (MHSA) into individually comprehensible components. Lorsa is desi…

Dictionary Learning

Transcoders Find Interpretable LLM Feature Circuits

2024-06-17 · Jacob Dunefsky, Philippe Chlenski, Neel Nanda

A key goal in mechanistic interpretability is circuit analysis: finding sparse subgraphs of models corresponding to specific behaviors or capabilities. However, MLP sublayers make fine-grained circuit analysis on transfo…