Learning activation functions from data using cubic spline interpolation
Neural networks require a careful design in order to perform properly on a given task. In particular, selecting a good activation function (possibly in a data-dependent fashion) is a crucial step, which remains an open problem in the research community. Despite a large amount of investigations, most current implementations simply select one fixed function from a small set of candidates, which is not adapted during training, and is shared among all neurons throughout the different layers. However, neither two of these assumptions can be supposed optimal in practice. In this paper, we present a principled way to have data-dependent adaptation of the activation functions, which is performed independently for each neuron. This is achieved by leveraging over past and present advances on cubic spline interpolation, allowing for local adaptation of the functions around their regions of use. The resulting algorithm is relatively cheap to implement, and overfitting is counterbalanced by the inclusion of a novel damping criterion, which penalizes unwanted oscillations from a predefined shape. Experimental results validate the proposal over two well-known benchmarks.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Cubic Spline Smoothing Compensation for Irregularly Sampled Sequences
The marriage of recurrent neural networks and neural ordinary differential networks (ODE-RNN) is effective in modeling irregularly-observed sequences. While ODE produces the smooth hidden states between observation inter…
Attention is a smoothed cubic spline
We highlight a perhaps important but hitherto unobserved insight: The attention module in a transformer is a smoothed cubic spline. Viewed in this manner, this mysterious but critical component of a transformer becomes a…
DecoderCubic Spline Interpolation Segmenting over Conventional Segmentation Procedures: Application and Advantages
To design a novel method for segmenting the image using Cubic Spline Interpolation and compare it with different techniques to determine which gives an efficient data to segment an image. This paper compares polynomial l…
Image SegmentationSemantic SegmentationApplication Research of Spline Interpolation and ARIMA in the Field of Stock Market Forecasting
The ARIMA (Autoregressive Integrated Moving Average model) has extensive applications in the field of time series forecasting. However, the predictive performance of the ARIMA model is limited when dealing with data gaps…
Time SeriesTime Series ForecastingImplicit Bias of Gradient Descent for Mean Squared Error Regression with Two-Layer Wide Neural Networks
We investigate gradient descent training of wide neural networks and the corresponding implicit bias in function space. For univariate regression, we show that the solution of training a width-$n$ shallow ReLU network is…
regression