ErfAct and Pserf: Non-monotonic Smooth Trainable Activation Functions

2021-09-09 · Koushik Biswas, Sandeep Kumar, Shilpak Banerjee, Ashish Kumar Pandey

An activation function is a crucial component of a neural network that introduces non-linearity in the network. The state-of-the-art performance of a neural network depends also on the perfect choice of an activation function. We propose two novel non-monotonic smooth trainable activation functions, called ErfAct and Pserf. Experiments suggest that the proposed functions improve the network performance significantly compared to the widely used activations like ReLU, Swish, and Mish. Replacing ReLU by ErfAct and Pserf, we have 5.68% and 5.42% improvement for top-1 accuracy on Shufflenet V2 (2.0x) network in CIFAR100 dataset, 2.11% and 1.96% improvement for top-1 accuracy on Shufflenet V2 (2.0x) network in CIFAR10 dataset, 1.0%, and 1.0% improvement on mean average precision (mAP) on SSD300 model in Pascal VOC dataset.

📄 PDF Abstract BibTeX arXiv:2109.04386

Code (0)

등록된 구현이 없습니다.

Methods 이 논문이 사용한 방법론

ReLU How Do I Communicate to Expedia? How Do I Communicate to Expedia? – Call ☎️ +1-(888) 829 (0881) or +1-805-330-4056 or +1-805-330-4056 for Live Support & Special Travel…

Grouped Convolution A Grouped Convolution uses a group of convolutions - multiple kernels per layer - resulting in multiple channel outputs per layer. This leads to wider networks helping a…

Channel Shuffle 설명 없음

Groupwise Point Convolution 설명 없음

Batch Normalization 설명 없음

Pointwise Convolution Pointwise Convolution is a type of convolution that uses a 1x1 kernel: a kernel that iterates through every single point. This…

Average Pooling 설명 없음

Depthwise Convolution Depthwise Convolution is a type of convolution where we apply a single convolutional filter for each input channel. In the regular 2D…

ErfReLU: Adaptive Activation Function for Deep Neural Network

2023-06-02 · Ashish Rajanand, Pradeep Singh

Recent research has found that the activation function (AF) selected for adding non-linearity into the output can have a big impact on how effectively deep learning networks perform. Developing activation functions that …

Expanded Gating Ranges Improve Activation Functions

2024-05-25 · Allen Hao Huang

Activation functions are core components of all deep learning architectures. Currently, the most popular activation functions are smooth ReLU variants like GELU and SiLU. These are self-gated activation functions where t…

Stochastic Neural Networks with Monotonic Activation Functions

2016-01-01 · Siamak Ravanbakhsh, Barnabas Poczos, Jeff Schneider, Dale Schuurmans 외

We propose a Laplace approximation that creates a stochastic unit from any smooth monotonic activation function, using only Gaussian noise. This paper investigates the application of this stochastic approximation in trai…

A Non-monotonic Smooth Activation Function

2023-10-16 · Koushik Biswas, Meghana Karri, Ulaş Bağcı

Activation functions are crucial in deep learning models since they introduce non-linearity into the networks, allowing them to learn from errors and make adjustments, which is essential for learning complex patterns. Th…

Adversarial Attack Adversarial Robustness image-classification Image Classification+2

A Counterfactual Analysis of the Dishonest Casino

2024-05-24 · Martin Haugh, Raghav Singal

The dishonest casino is a well-known hidden Markov model (HMM) used in educational settings to introduce HMMs and graphical models. Here, a sequence of die rolls is observed, with the casino switching between a fair and …

Causal Inference counterfactual Counterfactual Inference