ARiA: Utilizing Richard's Curve for Controlling the Non-monotonicity of the Activation Function in Deep Neural Nets
This work introduces a novel activation unit that can be efficiently employed in deep neural nets (DNNs) and performs significantly better than the traditional Rectified Linear Units (ReLU). The function developed is a two parameter version of the specialized Richard's Curve and we call it Adaptive Richard's Curve weighted Activation (ARiA). This function is non-monotonous, analogous to the newly introduced Swish, however allows a precise control over its non-monotonous convexity by varying the hyper-parameters. We first demonstrate the mathematical significance of the two parameter ARiA followed by its application to benchmark problems such as MNIST, CIFAR-10 and CIFAR-100, where we compare the performance with ReLU and Swish units. Our results illustrate a significantly superior performance on all these datasets, making ARiA a potential replacement for ReLU and other activations in DNNs.
Code (0)
등록된 구현이 없습니다.
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
On Learning-Curve Monotonicity for Maximum Likelihood Estimators
The property of learning-curve monotonicity, highlighted in a recent series of work by Loog, Mey and Viering, describes algorithms which only improve in average performance given more data, for any underlying data distri…
Minimizers of the Empirical Risk and Risk Monotonicity
Plotting a learner's average performance against the number of training samples results in a learning curve. Studying such curves on one or more data sets is a way to get to a better understanding of the generalization p…
Density Estimation${\mathcal K}$-monotonicity and feedback synthesis for incrementally stable networks
We discuss the role of monotonicity in enabling numerically tractable modular control design for networked nonlinear systems. We first show that the variational systems of monotone systems can be embedded into positive s…
Exploiting Monotonicity to Design an Adaptive PI Passivity-Based Controller for a Fuel-Cell System
We present a controller for a power electronic system composed of a fuel cell (FC) connected to a boost converter which feeds a resistive load. The controller aims to regulate the output voltage of the converter regardle…
Evaluating undercounts in epidemics: response to Maruotti et al. 2022
Maruotti et al. 2022 used a mark-recapture approach to estimate bounds on the true number of monkeypox infections in various countries. These approaches are fundamentally flawed; it is impossible to estimate undercountin…