Kafnets: kernel-based non-parametric activation functions for neural networks

2017-07-13 · Simone Scardapane, Steven Van Vaerenbergh, Simone Totaro, Aurelio Uncini

Neural networks are generally built by interleaving (adaptable) linear layers with (fixed) nonlinear activation functions. To increase their flexibility, several authors have proposed methods for adapting the activation functions themselves, endowing them with varying degrees of flexibility. None of these approaches, however, have gained wide acceptance in practice, and research in this topic remains open. In this paper, we introduce a novel family of flexible activation functions that are based on an inexpensive kernel expansion at every neuron. Leveraging over several properties of kernel-based models, we propose multiple variations for designing and initializing these kernel activation functions (KAFs), including a multidimensional scheme allowing to nonlinearly combine information from different paths in the network. The resulting KAFs can approximate any mapping defined over a subset of the real line, either convex or nonconvex. Furthermore, they are smooth over their entire domain, linear in their parameters, and they can be regularized using any known scheme, including the use of $\ell_1$ penalties to enforce sparseness. To the best of our knowledge, no other known model satisfies all these properties simultaneously. In addition, we provide a relatively complete overview on alternative techniques for adapting the activation functions, which is currently lacking in the literature. A large set of experiments validates our proposal.

📄 PDF Abstract BibTeX arXiv:1707.04035

Code (2)

ispamm/kernel-activation-functions 공식 구현 pytorch

shruti-jadon/Siamese-Network-for-One-shot-Learning pytorch

Methods 이 논문이 사용한 방법론

KAF A Kernel Activation Function is a non-parametric activation function defined as a one-dimensional kernel approximator: $$ f(s) = \sum_{i=1}^D \alpha_i \kappa( s, d_i)…

Improving Siamese Networks for One Shot Learning using Kernel Based Activation functions

2019-10-22 · Shruti Jadon, Aditya Acrot Srinivasan

The lack of a large amount of training data has always been the constraining factor in solving a lot of problems in machine learning, making One Shot Learning one of the most intriguing ideas in machine learning. It aims…

BIG-bench Machine Learning Deep Learning One-Shot Learning

Complex-valued Neural Networks with Non-parametric Activation Functions

2018-02-22 · Simone Scardapane, Steven Van Vaerenbergh, Amir Hussain, Aurelio Uncini

Complex-valued neural networks (CVNNs) are a powerful modeling tool for domains where data can be naturally interpreted in terms of complex numbers. However, several analytical properties of the complex domain (e.g., hol…

On the Stability and Generalization of Learning with Kernel Activation Functions

2019-03-28 · Michele Cirillo, Simone Scardapane, Steven Van Vaerenbergh, Aurelio Uncini

In this brief we investigate the generalization properties of a recently-proposed class of non-parametric activation functions, the kernel activation functions (KAFs). KAFs introduce additional parameters in the learning…

Multikernel activation functions: formulation and a case study

2019-01-29 · Simone Scardapane, Elena Nieddu, Donatella Firmani, Paolo Merialdo

The design of activation functions is a growing research area in the field of neural networks. In particular, instead of using fixed point-wise functions (e.g., the rectified linear unit), several authors have proposed w…

Optical Character Recognition (OCR)

Improving Graph Convolutional Networks with Non-Parametric Activation Functions

2018-02-26 · Simone Scardapane, Steven Van Vaerenbergh, Danilo Comminiello, Aurelio Uncini

Graph neural networks (GNNs) are a class of neural networks that allow to efficiently perform inference on data that is associated to a graph structure, such as, e.g., citation networks or knowledge graphs. While several…

Knowledge Graphs