Parametric Exponential Linear Unit for Deep Convolutional Neural Networks

2016-05-30 · Ludovic Trottier, Philippe Giguère, Brahim Chaib-Draa

Object recognition is an important task for improving the ability of visual systems to perform complex scene understanding. Recently, the Exponential Linear Unit (ELU) has been proposed as a key component for managing bias shift in Convolutional Neural Networks (CNNs), but defines a parameter that must be set by hand. In this paper, we propose learning a parameterization of ELU in order to learn the proper activation shape at each layer in the CNNs. Our results on the MNIST, CIFAR-10/100 and ImageNet datasets using the NiN, Overfeat, All-CNN and ResNet networks indicate that our proposed Parametric ELU (PELU) has better performances than the non-parametric ELU. We have observed as much as a 7.28% relative error improvement on ImageNet with the NiN network, with only 0.0003% parameter increase. Our visual examination of the non-linear behaviors adopted by Vgg using PELU shows that the network took advantage of the added flexibility by learning different activations at different layers.

📄 PDF Abstract BibTeX arXiv:1605.09332

Code (0)

등록된 구현이 없습니다.

Tasks

Object Recognition Scene Understanding

Methods 이 논문이 사용한 방법론

Average Pooling 설명 없음

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

1x1 Convolution A 1 x 1 Convolution is a convolution with some special properties in that it can be used for dimensionality reduction,…

Batch Normalization 설명 없음

Bottleneck Residual Block A Bottleneck Residual Block is a variant of the residual block that utilises 1x1 convolutions to create a bottleneck. The…

Global Average Pooling Global Average Pooling is a pooling operation designed to replace fully connected layers in classical CNNs. The idea is to generate one feature map for each corresponding…

Residual Block Residual Blocks are skip-connection blocks that learn residual functions with reference to the layer inputs, instead of learning unreferenced functions. They were introduced…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Improving Deep Neural Network with Multiple Parametric Exponential Linear Units

2016-06-01 · Yang Li, Chunxiao Fan, Yong Li, Qiong Wu 외

Activation function is crucial to the recent successes of deep neural networks. In this paper, we first propose a new activation function, Multiple Parametric Exponential Linear Units (MPELU), aiming to generalize and un…

Robust Deep Network Learning of Nonlinear Regression Tasks by Parametric Leaky Exponential Linear Units (LELUs) and a Diffusion Metric

2025-07-09 · Enda D. V. Bigarella arxiv

This document proposes a parametric activation function (ac.f.) aimed at improving multidimensional nonlinear data regression. It is a established knowledge that nonlinear ac.f's are required for learning nonlinear datas…

Parametric Rectified Power Sigmoid Units: Learning Nonlinear Neural Transfer Analytical Forms

2021-01-25 · Abdourrahmane Mahamane Atto, Sylvie Galichet, Dominique Pastor, Nicolas Méger

The paper proposes representation functionals in a dual paradigm where learning jointly concerns both linear convolutional weights and parametric forms of nonlinear activation functions. The nonlinear forms proposed for …

BIG-bench Machine Learning

Parametric Variational Linear Units (PVLUs) in Deep Convolutional Networks

2021-10-23 · Aarush Gupta, Shikhar Ahuja

The Rectified Linear Unit is currently a state-of-the-art activation function in deep convolutional neural networks. To combat ReLU's dying neuron problem, we propose the Parametric Variational Linear Unit (PVLU), which …

Data Augmentation Transfer Learning

Empirical Evaluation of Rectified Activations in Convolutional Network

2015-05-05 · Bing Xu, Naiyan Wang, Tianqi Chen, Mu Li

In this paper we investigate the performance of different types of rectified activation functions in convolutional neural network: standard rectified linear unit (ReLU), leaky rectified linear unit (Leaky ReLU), parametr…

General Classification image-classification Image Classification