Learning Neural Activations
An artificial neuron is modelled as a weighted summation followed by an activation function which determines its output. A wide variety of activation functions such as rectified linear units (ReLU), leaky-ReLU, Swish, MISH, etc. have been explored in the literature. In this short paper, we explore what happens when the activation function of each neuron in an artificial neural network is learned natively from data alone. This is achieved by modelling the activation function of each neuron as a small neural network whose weights are shared by all neurons in the original network. We list our primary findings in the conclusions section. The code for our analysis is available at: https://github.com/amina01/Learning-Neural-Activations.
Code (2)
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Evaluating Synthetic Activations composed of SAE Latents in GPT-2
Sparse Auto-Encoders (SAEs) are commonly employed in mechanistic interpretability to decompose the residual stream into monosemantic SAE latents. Recent work demonstrates that perturbing a model's activations at an early…
SensitivityMassive Activations in Large Language Models
We observe an empirical phenomenon in Large Language Models (LLMs) -- very few activations exhibit significantly larger values than others (e.g., 100,000 times larger). We call them massive activations. First, we demonst…
A new fMRI data analysis method using cross validation: Negative BOLD responses may be the deactivations of interneurons
Although functional magnetic resonance imaging (fMRI) is widely used for the study of brain functions, the blood oxygenation level dependent (BOLD) effect is incompletely understood. Particularly, negative BOLD responses…
Not All Diffusion Model Activations Have Been Evaluated as Discriminative Features
Diffusion models are initially designed for image generation. Recent research shows that the internal signals within their backbones, named activations, can also serve as dense features for various discriminative tasks s…
Allfeature selectionImage GenerationSemantic SegmentationGAS: Generative Activation-Aided Asynchronous Split Federated Learning
Split Federated Learning (SFL) splits and collaboratively trains a shared model between clients and server, where clients transmit activations and client-side models to server for updates. Recent SFL studies assume synch…
Federated Learning