Empirical analysis of non-linear activation functions for Deep Neural Networks in classification tasks
We provide an overview of several non-linear activation functions in a neural network architecture that have proven successful in many machine learning applications. We conduct an empirical analysis on the effectiveness of using these function on the MNIST classification task, with the aim of clarifying which functions produce the best results overall. Based on this first set of results, we examine the effects of building deeper architectures with an increasing number of hidden layers. We also survey the impact of using, on the same task, different initialisation schemes for the weights of our neural network. Using these sets of experiments as a base, we conclude by providing a optimal neural network architecture that yields impressive results in accuracy on the MNIST classification task.
Code (0)
등록된 구현이 없습니다.
Tasks
General ClassificationSimilar Papers 제목 키워드 기반
An Analysis of State-of-the-art Activation Functions For Supervised Deep Neural Network
This paper provides an analysis of state-of-the-art activation functions with respect to supervised classification of deep neural network. These activation functions comprise of Rectified Linear Units (ReLU), Exponential…
Acoustic Scene ClassificationClassificationGeneral ClassificationScene ClassificationEmpirical Loss Landscape Analysis of Neural Network Activation Functions
Activation functions play a significant role in neural network design by enabling non-linearity. The choice of activation function was previously shown to influence the properties of the resulting loss landscape. Underst…
On Output Activation Functions for Adversarial Losses: A Theoretical Analysis via Variational Divergence Minimization and An Empirical Study on MNIST Classification
Recent years have seen adversarial losses been applied to many fields. Their applications extend beyond the originally proposed generative modeling to conditional generative and discriminative settings. While prior work …
On the Importance of Normalisation Layers in Deep Learning with Piecewise Linear Activation Units
Deep feedforward neural networks with piecewise linear activations are currently producing the state-of-the-art results in several public datasets. The combination of deep learning models and piecewise linear activation …
General ClassificationImage ClassificationSemi-Periodic Activation for Time Series Classification
This paper investigates the lack of research on activation functions for neural network models in time series tasks. It highlights the need to identify essential properties of these activations to improve their effective…
ClassificationTime SeriesTime Series Classification