Variations on the Chebyshev-Lagrange Activation Function
We seek to improve the data efficiency of neural networks and present novel implementations of parameterized piece-wise polynomial activation functions. The parameters are the y-coordinates of n+1 Chebyshev nodes per hidden unit and Lagrangian interpolation between the nodes produces the polynomial on [-1, 1]. We show results for different methods of handling inputs outside [-1, 1] on synthetic datasets, finding significant improvements in capacity of expression and accuracy of interpolation in models that compute some form of linear extrapolation from either ends. We demonstrate competitive or state-of-the-art performance on the classification of images (MNIST and CIFAR-10) and minimally-correlated vectors (DementiaBank) when we replace ReLU or tanh with linearly extrapolated Chebyshev-Lagrange activations in deep residual architectures.
Code (2)
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
chebgreen: Learning and Interpolating Continuous Empirical Green's Functions from Data
In this work, we present a mesh-independent, data-driven library, chebgreen, to mathematically model one-dimensional systems, possessing an associated control parameter, and whose governing partial differential equation …
PointNet with KAN versus PointNet with MLP for 3D Classification and Segmentation of Point Sets
Kolmogorov-Arnold Networks (KANs) have recently gained attention as an alternative to traditional Multilayer Perceptrons (MLPs) in deep learning frameworks. KANs have been integrated into various deep learning architectu…
3D Classification3D Object Classification3D Point Cloud ClassificationKolmogorov-Arnold Networks+1Deterministic Reservoir Computing for Chaotic Time Series Prediction
Reservoir Computing was shown in recent years to be useful as efficient to learn networks in the field of time series tasks. Their randomized initialization, a computational benefit, results in drawbacks in theoretical a…
PredictionTime SeriesTime Series ForecastingTime Series PredictionPhysics-Informed Chebyshev Polynomial Neural Operator for Parametric Partial Differential Equations
Neural operators have emerged as powerful deep learning frameworks for approximating solution operators of parameterized partial differential equations (PDE). However, current methods predominantly rely on multilayer per…
Deep Neural Networks and Finite Elements of Any Order on Arbitrary Dimensions
In this study, we establish that deep neural networks employing ReLU and ReLU$^2$ activation functions can effectively represent Lagrange finite element functions of any order on various simplicial meshes in arbitrary di…