Universal Approximation Properties for an ODENet and a ResNet: Mathematical Analysis and Numerical Experiments
We prove a universal approximation property (UAP) for a class of ODENet and a class of ResNet, which are simplified mathematical models for deep learning systems with skip connections. The UAP can be stated as follows. Let $n$ and $m$ be the dimension of input and output data, and assume $m\leq n$. Then we show that ODENet of width $n+m$ with any non-polynomial continuous activation function can approximate any continuous function on a compact subset on $\mathbb{R}^n$. We also show that ResNet has the same property as the depth tends to infinity. Furthermore, we derive the gradient of a loss function explicitly with respect to a certain tuning variable. We use this to construct a learning algorithm for ODENet. To demonstrate the usefulness of this algorithm, we apply it to a regression problem, a binary classification, and a multinomial classification in MNIST.
Code (0)
등록된 구현이 없습니다.
Tasks
Binary ClassificationGeneral ClassificationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Universal approximation property of ODENet and ResNet with a single activation function
We study a universal approximation property of ODENet and ResNet. The ODENet is a map from an initial value to the final value of an ODE system in a finite interval. It is considered a mathematical model of a ResNet-type…
Accelerating ODE-Based Neural Networks on Low-Cost FPGAs
ODENet is a deep neural network architecture in which a stacking structure of ResNet is implemented with an ordinary differential equation (ODE) solver. It can reduce the number of parameters and strike a balance between…
Approximation Properties of Deep ReLU CNNs
This paper focuses on establishing $L^2$ approximation properties for deep ReLU convolutional neural networks (CNNs) in two-dimensional space. The analysis is based on a decomposition theorem for convolutional kernels wi…
Neural Flow Operators can Approximate any Operator: Abstract Frameworks and Universal Approximations
We introduce an abstract neural flow framework for neural networks and neural operators. The framework contains two continuous-depth models, namely neural flows with composition and separation structures, and covers both…
The Influence of the Memory Capacity of Neural DDEs on the Universal Approximation Property
Neural Ordinary Differential Equations (Neural ODEs), which are the continuous-time analog of Residual Neural Networks (ResNets), have gained significant attention in recent years. Similarly, Neural Delay Differential Eq…