A Non-monotonic Smooth Activation Function

2023-10-16 · Koushik Biswas, Meghana Karri, Ulaş Bağcı

Activation functions are crucial in deep learning models since they introduce non-linearity into the networks, allowing them to learn from errors and make adjustments, which is essential for learning complex patterns. The essential purpose of activation functions is to transform unprocessed input signals into significant output activations, promoting information transmission throughout the neural network. In this study, we propose a new activation function called Sqish, which is a non-monotonic and smooth function and an alternative to existing ones. We showed its superiority in classification, object detection, segmentation tasks, and adversarial robustness experiments. We got an 8.21% improvement over ReLU on the CIFAR100 dataset with the ShuffleNet V2 model in the FGSM adversarial attack. We also got a 5.87% improvement over ReLU on image classification on the CIFAR100 dataset with the ShuffleNet V2 model.

📄 PDF Abstract BibTeX arXiv:2310.10126

Code (0)

등록된 구현이 없습니다.

Tasks

Adversarial Attack Adversarial Robustness image-classification Image Classification object-detection Object Detection

Methods 이 논문이 사용한 방법론

ReLU How Do I Communicate to Expedia? How Do I Communicate to Expedia? – Call ☎️ +1-(888) 829 (0881) or +1-805-330-4056 or +1-805-330-4056 for Live Support & Special Travel…

Depthwise Convolution Depthwise Convolution is a type of convolution where we apply a single convolutional filter for each input channel. In the regular 2D…

Pointwise Convolution Pointwise Convolution is a type of convolution that uses a 1x1 kernel: a kernel that iterates through every single point. This…

Grouped Convolution A Grouped Convolution uses a group of convolutions - multiple kernels per layer - resulting in multiple channel outputs per layer. This leads to wider networks helping a…

Average Pooling 설명 없음

Convolution A convolution is a type of matrix operation, consisting of a kernel, a small matrix of weights, that slides over input data performing element-wise multiplication with the…

ShuffleNet V2 Block ShuffleNet V2 Block is an image model block used in the ShuffleNet V2 architecture, where speed is the metric optimized for…

Channel Shuffle 설명 없음

ErfAct and Pserf: Non-monotonic Smooth Trainable Activation Functions

2021-09-09 · Koushik Biswas, Sandeep Kumar, Shilpak Banerjee, Ashish Kumar Pandey

An activation function is a crucial component of a neural network that introduces non-linearity in the network. The state-of-the-art performance of a neural network depends also on the perfect choice of an activation fun…

Stochastic Neural Networks with Monotonic Activation Functions

2016-01-01 · Siamak Ravanbakhsh, Barnabas Poczos, Jeff Schneider, Dale Schuurmans 외

We propose a Laplace approximation that creates a stochastic unit from any smooth monotonic activation function, using only Gaussian noise. This paper investigates the application of this stochastic approximation in trai…

Saturated Non-Monotonic Activation Functions

2023-05-12 · Junjia Chen, Zhibin Pan

Activation functions are essential to deep learning networks. Popular and versatile activation functions are mostly monotonic functions, some non-monotonic activation functions are being explored and show promising perfo…

Deep Learning image-classification Image Classification

Expanded Gating Ranges Improve Activation Functions

2024-05-25 · Allen Hao Huang

Activation functions are core components of all deep learning architectures. Currently, the most popular activation functions are smooth ReLU variants like GELU and SiLU. These are self-gated activation functions where t…

Swish-T : Enhancing Swish Activation with Tanh Bias for Improved Neural Network Performance

2024-07-01 · Youngmin Seo, Jinha Kim, Unsang Park

We propose the Swish-T family, an enhancement of the existing non-monotonic activation function Swish. Swish-T is defined by adding a Tanh bias to the original Swish function. This modification creates a family of Swish-…