P-Swish: Activation Function with Learnable Parameters Based on Swish Activation Function in Deep Learning
In order to improve the performance of a deep neural network, the activation function is an important aspect that we must research continuously, that is why we have expanded the research in this direction. We introduced a novel P-Swish activation function (Parametric Swish), which is able to bring performance improvements on object classification tasks using datasets such as CIFAR-10, CIFAR-100, but we will see that we also used datasets for Natural Language Processing (NLP). To test it, we used several types of architectures, including LeNet-5, Network in Network (NiN), and ResNet34 compared to popular activation functions such as sigmoid, ReLU, Swish, and our proposals. In particular, the P-Swish function facilitates fast network training, which makes it suitable for the Transfer Learning technique.
Code (0)
등록된 구현이 없습니다.
Tasks
Image ClassificationTransfer LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
SG-Blend: Learning an Interpolation Between Improved Swish and GELU for Robust Neural Representations
The design of activation functions remains a pivotal component in optimizing deep neural networks. While prevailing choices like Swish and GELU demonstrate considerable efficacy, they often exhibit domain-specific optima…
E-swish: Adjusting Activations to Different Network Depths
Activation functions have a notorious impact on neural networks on both training and testing the models against the desired problem. Currently, the most used activation function is the Rectified Linear Unit (ReLU). This …
Swish-T : Enhancing Swish Activation with Tanh Bias for Improved Neural Network Performance
We propose the Swish-T family, an enhancement of the existing non-monotonic activation function Swish. Swish-T is defined by adding a Tanh bias to the original Swish function. This modification creates a family of Swish-…
Tangma: A Tanh-Guided Activation Function with Learnable Parameters
Activation functions are key to effective backpropagation and expressiveness in deep neural networks. This work introduces Tangma, a new activation function that combines the smooth shape of the hyperbolic tangent with t…
SwishReLU: A Unified Approach to Activation Functions for Enhanced Deep Neural Networks Performance
ReLU, a commonly used activation function in deep neural networks, is prone to the issue of "Dying ReLU". Several enhanced versions, such as ELU, SeLU, and Swish, have been introduced and are considered to be less common…