SAU: Smooth activation function using convolution with approximate identities

2021-09-27 · Koushik Biswas, Sandeep Kumar, Shilpak Banerjee, Ashish Kumar Pandey

Well-known activation functions like ReLU or Leaky ReLU are non-differentiable at the origin. Over the years, many smooth approximations of ReLU have been proposed using various smoothing techniques. We propose new smooth approximations of a non-differentiable activation function by convolving it with approximate identities. In particular, we present smooth approximations of Leaky ReLU and show that they outperform several well-known activation functions in various datasets and models. We call this function Smooth Activation Unit (SAU). Replacing ReLU by SAU, we get 5.12% improvement with ShuffleNet V2 (2.0x) model on CIFAR100 dataset.

📄 PDF Abstract BibTeX arXiv:2109.13210

Code (0)

등록된 구현이 없습니다.

Methods 이 논문이 사용한 방법론

ReLU How Do I Communicate to Expedia? How Do I Communicate to Expedia? – Call ☎️ +1-(888) 829 (0881) or +1-805-330-4056 or +1-805-330-4056 for Live Support & Special Travel…

Pointwise Convolution Pointwise Convolution is a type of convolution that uses a 1x1 kernel: a kernel that iterates through every single point. This…

Grouped Convolution A Grouped Convolution uses a group of convolutions - multiple kernels per layer - resulting in multiple channel outputs per layer. This leads to wider networks helping a…

Groupwise Point Convolution 설명 없음

Batch Normalization 설명 없음

Residual Connection 설명 없음

Average Pooling 설명 없음

Depthwise Convolution Depthwise Convolution is a type of convolution where we apply a single convolutional filter for each input channel. In the regular 2D…

ConvNets with Smooth Adaptive Activation Functions for Regression

2017-01-01 · Le Hou ; Dimitris Samaras ; Tahsin M. Kurc ; Yi Gao ; Joel H. Saltz

Within Neural Networks (NN), the parameters of Adaptive Activation Functions (AAF) control the shapes of activation functions. These parameters are trained along with other parameters in the NN. AAFs have improved perfo…

Age And Gender Classification Pose Estimation regression

Provable wavelet-based neural approximation

2025-04-23 · Youngmi Hur, Hyojae Lim, Mikyoung Lim

In this paper, we develop a wavelet-based theoretical framework for analyzing the universal approximation capabilities of neural networks over a wide range of activation functions. Leveraging wavelet frame theory on the …

Rational neural networks

2020-04-04 · NeurIPS 2020 12 · Nicolas Boullé, Yuji Nakatsukasa, Alex Townsend

We consider neural networks with rational activation functions. The choice of the nonlinear activation function in deep learning architectures is crucial and heavily impacts the performance of a neural network. We establ…

Smooth function approximation by deep neural networks with general activation functions

2019-06-17 · Ilsang Ohn, Yongdai Kim

There has been a growing interest in expressivity of deep neural networks. However, most of the existing work about this topic focuses only on the specific activation function such as ReLU or sigmoid. In this paper, we i…

General Classification

Low Curvature Activations Reduce Overfitting in Adversarial Training

2021-02-15 · ICCV 2021 10 · Vasu Singla, Sahil Singla, David Jacobs, Soheil Feizi

Adversarial training is one of the most effective defenses against adversarial attacks. Previous works suggest that overfitting is a dominant phenomenon in adversarial training leading to a large generalization gap betwe…