Learning Activation Functions to Improve Deep Neural Networks
Artificial neural networks typically have a fixed, non-linear activation function at each neuron. We have designed a novel form of piecewise linear activation function that is learned independently for each neuron using gradient descent. With this adaptive activation function, we are able to improve upon deep neural network architectures composed of static rectified linear units, achieving state-of-the-art performance on CIFAR-10 (7.51%), CIFAR-100 (30.83%), and a benchmark from high-energy physics involving Higgs boson decay modes.
Code (3)
Tasks
Image ClassificationSimilar Papers 제목 키워드 기반
Expanded Gating Ranges Improve Activation Functions
Activation functions are core components of all deep learning architectures. Currently, the most popular activation functions are smooth ReLU variants like GELU and SiLU. These are self-gated activation functions where t…
Frequency and Generalisation of Periodic Activation Functions in Reinforcement Learning
Periodic activation functions, often referred to as learned Fourier features have been widely demonstrated to improve sample efficiency and stability in a variety of deep RL algorithms. Potentially incompatible hypothese…
reinforcement-learningTanhSoft -- a family of activation functions combining Tanh and Softplus
Deep learning at its core, contains functions that are composition of a linear transformation with a non-linear function known as activation function. In past few years, there is an increasing interest in construction of…
General ClassificationImage ClassificationImproving Quaternion Neural Networks with Quaternionic Activation Functions
In this paper, we propose novel quaternion activation functions where we modify either the quaternion magnitude or the phase, as an alternative to the commonly used split activation functions. We define criteria that are…
image-classificationImage ClassificationBiologically Inspired Oscillating Activation Functions Can Bridge the Performance Gap between Biological and Artificial Neurons
The recent discovery of special human neocortical pyramidal neurons that can individually learn the XOR function highlights the significant performance gap between biological and artificial neurons. The output of these p…
Feature Engineering