From Taylor Series to Fourier Synthesis: The Periodic Linear Unit
The dominant paradigm in modern neural networks relies on simple, monotonically-increasing activation functions like ReLU. While effective, this paradigm necessitates large, massively-parameterized models to approximate complex functions. In this paper, we introduce the Periodic Linear Unit (PLU), a learnable sine-wave based activation with periodic non-monotonicity. PLU is designed for maximum expressive power and numerical stability, achieved through its formulation and a paired innovation we term Repulsive Reparameterization, which prevents the activation from collapsing into a non-expressive linear function. We demonstrate that a minimal MLP with only two PLU neurons can solve the spiral classification task, a feat impossible for equivalent networks using standard activations. This suggests a paradigm shift from networks as piecewise Taylor-like approximators to powerful Fourier-like function synthesizers, achieving exponential gains in parameter efficiency by placing intelligence in the neuron itself.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
A Relation Spectrum Inheriting Taylor Series: Muscle Synergy and Coupling for Hand
There are two famous function decomposition methods in math: Taylor Series and Fourier Series. Fourier series developed into Fourier spectrum, which was applied to signal decomposition\analysis. However, because the Tayl…
MathRelationA Fourier State Space Model for Bayesian ODE Filters
Gaussian ODE filtering is a probabilistic numerical method to solve ordinary differential equations (ODEs). It computes a Bayesian posterior over the solution from evaluations of the vector field defining the ODE. Its mo…
Index Set Fourier Series Features for Approximating Multi-dimensional Periodic Kernels
Periodicity is often studied in timeseries modelling with autoregressive methods but is less popular in the kernel literature, particularly for higher dimensional problems such as in textures, crystallography, and quantu…
Gaussian ProcessesWFTNet: Exploiting Global and Local Periodicity in Long-term Time Series Forecasting
Recent CNN and Transformer-based models tried to utilize frequency and periodicity information for long-term time series forecasting. However, most existing work is based on Fourier transform, which cannot capture fine-g…
Time SeriesTime Series ForecastingHarmonic analysis of the arctangent function regarding the angular error introduced by superimposed Fourier series for application in sine/cosine angle encoders
We present a rigorous analytical method for harmonic analysis of the angular error of rotary and linear encoders with sine/cosine output signals in quadrature that are distorted by superimposed Fourier series. To calcula…