Harmonics of Learning: Universal Fourier Features Emerge in Invariant Networks
In this work, we formally prove that, under certain conditions, if a neural network is invariant to a finite group then its weights recover the Fourier transform on that group. This provides a mathematical explanation for the emergence of Fourier features -- a ubiquitous phenomenon in both biological and artificial learning systems. The results hold even for non-commutative groups, in which case the Fourier transform encodes all the irreducible unitary group representations. Our findings have consequences for the problem of symmetry discovery. Specifically, we demonstrate that the algebraic structure of an unknown group can be recovered from the weights of a network that is at least approximately invariant within certain bounds. Overall, this work contributes to a foundation for an algebraic learning theory of invariant neural network representations.
Code (1)
Tasks
Learning TheorySimilar Papers 제목 키워드 기반
Optimal Motion Planning using Finite Fourier Series in a Learning-based Collision Field
This paper utilizes finite Fourier series to represent a time-continuous motion and proposes a novel planning method that adjusts the motion harmonics of each manipulator joint. Primarily, we sum the potential energy for…
Motion PlanningOptimal Motion PlanningA Functional Approach to Rotation Equivariant Non-Linearities for Tensor Field Networks.
Learning pose invariant representation is a fundamental problem in shape analysis. Most existing deep learning algorithms for 3D shape analysis are not robust to rotations and are often trained on synthetic datasets …
On the expressivity of embedding quantum kernels
One of the most natural connections between quantum and classical machine learning has been established in the context of kernel methods. Kernel methods rely on kernels, which are inner products of feature vectors living…
Enabling Efficient Equivariant Operations in the Fourier Basis via Gaunt Tensor Products
Developing equivariant neural networks for the E(3) group plays an important role in modeling 3D data across real-world applications. Enforcing this equivariance primarily involves the tensor products of irreducible repr…
Compressive Sensing Empirical Wavelet Transform for Frequency-Banded Power Measurement Considering Interharmonics
Power measurement algorithms based on Fourier transform are susceptible to errors caused by interharmonics, while wavelet transform algorithms are particularly sensitive to even harmonics due to band decomposition effect…
Compressive Sensing