Comparison of non-linear activation functions for deep neural networks on MNIST classification task
Activation functions play a key role in neural networks so it becomes fundamental to understand their advantages and disadvantages in order to achieve better performances. This paper will first introduce common types of non linear activation functions that are alternative to the well known sigmoid function and then evaluate their characteristics. Moreover deeper neural networks will be analysed because they positively influence the final performances compared to shallower networks. They also strictly depend on the weight initialisation hence the effect of drawing weights from Gaussian and uniform distribution will be analysed making particular attention on how the number of incoming and outgoing connection to a node influence the whole network.
Code (2)
Tasks
General ClassificationSimilar Papers 제목 키워드 기반
How important are activation functions in regression and classification? A survey, performance comparison, and future directions
Inspired by biological neurons, the activation functions play an essential part in the learning process of any artificial neural network commonly used in many real-world problems. Various activation functions have been p…
Physics-informed machine learningregressionAn Analysis of State-of-the-art Activation Functions For Supervised Deep Neural Network
This paper provides an analysis of state-of-the-art activation functions with respect to supervised classification of deep neural network. These activation functions comprise of Rectified Linear Units (ReLU), Exponential…
Acoustic Scene ClassificationClassificationGeneral ClassificationScene ClassificationEmpirical analysis of non-linear activation functions for Deep Neural Networks in classification tasks
We provide an overview of several non-linear activation functions in a neural network architecture that have proven successful in many machine learning applications. We conduct an empirical analysis on the effectiveness …
General ClassificationReprogrammable Electro-Optic Nonlinear Activation Functions for Optical Neural Networks
We introduce an electro-optic hardware platform for nonlinear activation functions in optical neural networks. The optical-to-optical nonlinearity operates by converting a small portion of the input optical signal into a…
Activation Function Optimization Scheme for Image Classification
Activation function has a significant impact on the dynamics, convergence, and performance of deep neural networks. The search for a consistent and high-performing activation function has always been a pursuit during dee…
Classificationimage-classificationImage Classification