TanhSoft -- a family of activation functions combining Tanh and Softplus
Deep learning at its core, contains functions that are composition of a linear transformation with a non-linear function known as activation function. In past few years, there is an increasing interest in construction of novel activation functions resulting in better learning. In this work, we propose a family of novel activation functions, namely TanhSoft, with four undetermined hyper-parameters of the form tanh({\alpha}x+{\beta}e^{{\gamma}x})ln({\delta}+e^x) and tune these hyper-parameters to obtain activation functions which are shown to outperform several well known activation functions. For instance, replacing ReLU with xtanh(0.6e^x)improves top-1 classification accuracy on CIFAR-10 by 0.46% for DenseNet-169 and 0.7% for Inception-v3 while with tanh(0.87x)ln(1 +e^x) top-1 classification accuracy on CIFAR-100 improves by 1.24% for DenseNet-169 and 2.57% for SimpleNet model.
Code (0)
등록된 구현이 없습니다.
Tasks
General ClassificationImage ClassificationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
ErfReLU: Adaptive Activation Function for Deep Neural Network
Recent research has found that the activation function (AF) selected for adding non-linearity into the output can have a big impact on how effectively deep learning networks perform. Developing activation functions that …
EIS -- a family of activation functions combining Exponential, ISRU, and Softplus
Activation functions play a pivotal role in the function learning using neural networks. The non-linearity in the learned function is achieved by repeated use of the activation function. Over the years, numerous activati…
Regularized Flexible Activation Function Combinations for Deep Neural Networks
Activation in deep neural networks is fundamental to achieving non-linear mappings. Traditional studies mainly focus on finding fixed activations for a particular set of learning tasks or model architectures. The researc…
Image CompressionPhilosophyTime SeriesTime Series Analysis+1Parametric Leaky Tanh: A New Hybrid Activation Function for Deep Learning
Activation functions (AFs) are crucial components of deep neural networks (DNNs), having a significant impact on their performance. An activation function in a DNN is typically a smooth, nonlinear function that transform…
Rectifier Neural Network with a Dual-Pathway Architecture for Image Denoising
Recently deep neural networks based on tanh activation function have shown their impressive power in image denoising. In this letter, we try to use rectifier function instead of tanh and propose a dual-pathway rectifier …
DenoisingImage Denoising