Competition-based Adaptive ReLU for Deep Neural Networks
Activation functions introduce nonlinearity into deep neural networks. Most popular activation functions allow positive values to pass through while blocking or suppressing negative values. From the idea that positive values and negative values are equally important, and they must compete for activation, we proposed a new Competition-based Adaptive ReLU (CAReLU). CAReLU scales the input values based on the competition results between positive values and negative values. It defines two parameters to adjust the scaling strategy and can be trained uniformly with other network parameters. We verify the effectiveness of CAReLU on image classification, super-resolution, and natural language processing tasks. In the experiment, our method performs better than other widely used activation functions. In the case of replacing ReLU in ResNet-18 with our proposed activation function, it improves the classification accuracy on the CIFAR-100 dataset. The effectiveness and the new perspective on the utilization of competition results between positive values and negative values make CAReLU a promising activation function.
Code (0)
등록된 구현이 없습니다.
Tasks
Blockingimage-classificationImage ClassificationSuper-ResolutionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
ErfReLU: Adaptive Activation Function for Deep Neural Network
Recent research has found that the activation function (AF) selected for adding non-linearity into the output can have a big impact on how effectively deep learning networks perform. Developing activation functions that …
How do noise tails impact on deep ReLU networks?
This paper investigates the stability of deep ReLU neural networks for nonparametric regression under the assumption that the noise has only a finite p-th moment. We unveil how the optimal rate of convergence depends on …
regressionBrownian ReLU(Br-ReLU): A New Activation Function for a Long-Short Term Memory (LSTM) Network
Deep learning models are effective for sequential data modeling, yet commonly used activation functions such as ReLU, LeakyReLU, and PReLU often exhibit gradient instability when applied to noisy, non-stationary financia…
Modality Competition: What Makes Joint Training of Multi-modal Network Fail in Deep Learning? (Provably)
Despite the remarkable success of deep multi-modal learning in practice, it has not been well-explained in theory. Recently, it has been observed that the best uni-modal network outperforms the jointly trained multi-moda…
Why North Korean Refugees are Reluctant to Compete: The Roles of Cognitive Ability
The study compares the competitiveness of three Korean groups raised in different institutional environments: South Korea, North Korea, and China. Laboratory experiments reveal that North Korean refugees are less likely …