Function Fitting Based on Kolmogorov-Arnold Theorem and Kernel Functions

2025-03-29 · Jianpeng Liu, Qizhi Pan

This paper proposes a unified theoretical framework based on the Kolmogorov-Arnold representation theorem and kernel methods. By analyzing the mathematical relationship among kernels, B-spline basis functions in Kolmogorov-Arnold Networks (KANs) and the inner product operation in self-attention mechanisms, we establish a kernel-based feature fitting framework that unifies the two models as linear combinations of kernel functions. Under this framework, we propose a low-rank Pseudo-Multi-Head Self-Attention module (Pseudo-MHSA), which reduces the parameter count of traditional MHSA by nearly 50\%. Furthermore, we design a Gaussian kernel multi-head self-attention variant (Gaussian-MHSA) to validate the effectiveness of nonlinear kernel functions in feature extraction. Experiments on the CIFAR-10 dataset demonstrate that Pseudo-MHSA model achieves performance comparable to the ViT model of the same dimensionality under the MAE framework and visualization analysis reveals their similarity of multi-head distribution patterns. Our code is publicly available.

📄 PDF Abstract BibTeX arXiv:2503.23038

Code (1)

cfrslyr/Experiments 공식 구현 pytorch

Tasks

Kolmogorov-Arnold Networks

Methods 이 논문이 사용한 방법론

MAE 설명 없음

The Kolmogorov-Arnold representation theorem revisited

2020-07-31 · Johannes Schmidt-Hieber

There is a longstanding debate whether the Kolmogorov-Arnold representation theorem can explain the use of more than one hidden layer in neural networks. The Kolmogorov-Arnold representation decomposes a multivariate fun…

Error bounds for deep ReLU networks using the Kolmogorov--Arnold superposition theorem

2019-06-27 · Hadrien Montanelli, Haizhao Yang

We prove a theorem concerning the approximation of multivariate functions by deep ReLU networks, for which the curse of the dimensionality is lessened. Our theorem is based on a constructive proof of the Kolmogorov--Arno…

A Practitioner's Guide to Kolmogorov-Arnold Networks

2025-10-28 · Amir Noorizadegan, Sifan Wang, Leevan Ling, Juan P. Dominguez-Morales arxiv

Kolmogorov-Arnold Networks (KANs), whose design is inspired-rather than dictated-by the Kolmogorov superposition theorem, have emerged as a structured alternative to MLPs. This review provides a systematic and comprehens…

Structured and Informed Probabilistic Modeling with the Thermodynamic Kolmogorov-Arnold Model

2025-06-17 · Prithvi Raj

We adapt the Kolmogorov-Arnold Representation Theorem to generative modeling by reinterpreting its inner functions as a Markov Kernel between probability spaces via inverse transform sampling. We present a generative mod…

Diversity

KKANs: Kurkova-Kolmogorov-Arnold Networks and Their Learning Dynamics

2024-12-21 · Juan Diego Toscano, Li-Lian Wang, George Em Karniadakis

Inspired by the Kolmogorov-Arnold representation theorem and Kurkova's principle of using approximate representations, we propose the Kurkova-Kolmogorov-Arnold Network (KKAN), a new two-block architecture that combines r…

Kolmogorov-Arnold Networks Operator learning Physics-informed machine learning