Why High-rank Neural Networks Generalize?: An Algebraic Framework with RKHSs
We derive a new Rademacher complexity bound for deep neural networks using Koopman operators, group representations, and reproducing kernel Hilbert spaces (RKHSs). The proposed bound describes why the models with high-rank weight matrices generalize well. Although there are existing bounds that attempt to describe this phenomenon, these existing bounds can be applied to limited types of models. We introduce an algebraic representation of neural networks and a kernel function to construct an RKHS to derive a bound for a wider range of realistic models. This work paves the way for the Koopman-based theory for Rademacher complexity bounds to be valid for more practical situations.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Spectral Truncation Kernels: Noncommutativity in $C^*$-algebraic Kernel Machines
$C^*$-algebra-valued kernels could pave the way for the next generation of kernel machines. To further our fundamental understanding of learning with $C^*$-algebraic kernels, we propose a new class of positive definite k…
Learning in RKHM: a $C^*$-Algebraic Twist for Kernel Machines
Supervised learning in reproducing kernel Hilbert space (RKHS) and vector-valued RKHS (vvRKHS) has been investigated for more than 30 years. In this paper, we provide a new twist to this rich literature by generalizing s…
Identifiability of an X-rank decomposition of polynomial maps
In this paper, we study a polynomial decomposition model that arises in problems of system identification, signal processing and machine learning. We show that this decomposition is a special case of the X-rank decomposi…
BIG-bench Machine LearningNonparametric Control Koopman Operators
This paper presents a novel Koopman (composition) operator representation framework for control systems in reproducing kernel Hilbert spaces (RKHSs) that is free of explicit dictionary or input parametrizations. By estab…
Learning TheoryModel Predictive ControlOperator learningExperimental Design for Linear Functionals in Reproducing Kernel Hilbert Spaces
Optimal experimental design seeks to determine the most informative allocation of experiments to infer an unknown statistical quantity. In this work, we investigate the optimal design of experiments for {\em estimation o…
Experimental Design