Sparse Autoencoders Trained on the Same Data Learn Different Features

2025-01-28 · Gonçalo Paulo, Nora Belrose

Sparse autoencoders (SAEs) are a useful tool for uncovering human-interpretable features in the activations of large language models (LLMs). While some expect SAEs to find the true underlying features used by a model, our research shows that SAEs trained on the same model and data, differing only in the random seed used to initialize their weights, identify different sets of features. For example, in an SAE with 131K latents trained on a feedforward network in Llama 3 8B, only 30% of the features were shared across different seeds. We observed this phenomenon across multiple layers of three different LLMs, two datasets, and several SAE architectures. While ReLU SAEs trained with the L1 sparsity loss showed greater stability across seeds, SAEs using the state-of-the-art TopK activation function were more seed-dependent, even when controlling for the level of sparsity. Our results suggest that the set of features uncovered by an SAE should be viewed as a pragmatically useful decomposition of activation space, rather than an exhaustive and universal list of features "truly used" by the model.

📄 PDF Abstract BibTeX arXiv:2501.16615

Code (0)

등록된 구현이 없습니다.

Methods 이 논문이 사용한 방법론

ReLU How Do I Communicate to Expedia? How Do I Communicate to Expedia? – Call ☎️ +1-(888) 829 (0881) or +1-805-330-4056 or +1-805-330-4056 for Live Support & Special Travel…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

LLaMA LLaMA is a collection of foundation language models ranging from 7B to 65B parameters. It is based on the transformer architecture with various improvements that were…

SET Dynamic Sparse Training method where weight mask is updated randomly periodically

Feedforward Network A Feedforward Network, or a Multilayer Perceptron (MLP), is a neural network with solely densely connected layers. This is the classic neural network architecture of the…

Image Compression: Sparse Coding vs. Bottleneck Autoencoders

2017-10-26 · Yijing Watkins, Mohammad Sayeh, Oleksandr Iaroshenko, Garrett Kenyon

Bottleneck autoencoders have been actively researched as a solution to image compression tasks. However, we observed that bottleneck autoencoders produce subjectively low quality reconstructed images. In this work, we ex…

General Classification Image Compression SSIM

Transcoders Beat Sparse Autoencoders for Interpretability

2025-01-31 · Gonçalo Paulo, Stepan Shabalin, Nora Belrose

Sparse autoencoders (SAEs) extract human-interpretable features from deep neural networks by transforming their activations into a sparse, higher dimensional latent space, and then reconstructing the activations from the…

ReSAE: Residualized Sparse Autoencoders for Multi-Layer Transformer Interventions

2026-05-27 · Prathyush Poduval, Calvin Yeung, Neel Desai, Mohsen Imani arxiv

Sparse autoencoders are usually trained one layer at a time, even though transformer residual stream activations are strongly coupled across depth. This creates a practical problem for multi-layer interventions: differen…

A new Sparse Auto-encoder based Framework using Grey Wolf Optimizer for Data Classification Problem

2022-01-29 · Ahmad Mozaffer Karim

One of the most important properties of deep auto-encoders (DAEs) is their capability to extract high level features from row data. Hence, especially recently, the autoencoders are preferred to be used in various classif…

Computer Security

Are We Using Autoencoders in a Wrong Way?

2023-09-04 · Gabriele Martino, Davide Moroni, Massimo Martinelli

Autoencoders are certainly among the most studied and used Deep Learning models: the idea behind them is to train a model in order to reconstruct the same input data. The peculiarity of these models is to compress the in…

Anomaly Detection Classification Denoising Dimensionality Reduction