Lattice-based Deep Neural Networks: Regularity and Tailored Regularization
This survey article is concerned with the application of lattice rules to Deep Neural Networks (DNNs), lattice rules being a family of quasi-Monte Carlo methods. They have demonstrated effectiveness in various contexts for high-dimensional integration and function approximation. They are extremely easy to implement thanks to their very simple formulation -- all that is required is a good integer generating vector of length matching the dimensionality of the problem. In recent years there has been a burst of research activities on the application and theory of DNNs. We review our recent article on using lattice rules as training points for DNNs with a smooth activation function, where we obtained explicit regularity bounds of the DNNs. By imposing restrictions on the network parameters to match the regularity features of the target function, we prove that DNNs with tailored lattice training points can achieve good theoretical generalization error bounds, with implied constants independent of the input dimension. We also demonstrate numerically that DNNs trained with our tailored regularization perform significantly better than with standard $\ell_2$ regularization.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Fourier Neural Operators with rank-1 lattice points and hyperbolic cross
The \emph{Fourier neural operator} (FNO) is a neural network architecture that learns mappings between function spaces. Its efficient implementation is based on the multi-dimensional Fourier transform. By deriving genera…
Regularity-Driven Facade Matching Between Aerial and Street Views
We present an approach for detecting and matching building facades between aerial view and street-view images. We exploit the regularity of urban scene facades as captured by their lattice structures and deduced from med…
Neural Network Training Techniques Regularize Optimization Trajectory: An Empirical Study
Modern deep neural network (DNN) trainings utilize various training techniques, e.g., nonlinear activation functions, batch normalization, skip-connections, etc. Despite their effectiveness, it is still mysterious how th…
Differentiable graph-structured models for inverse design of lattice materials
Architected materials possessing physico-chemical properties adaptable to disparate environmental conditions embody a disruptive new domain of materials science. Fueled by advances in digital design and fabrication, mate…
Graph Neural NetworkDeep neural networks for choice analysis: Enhancing behavioral regularity with gradient regularization
Deep neural networks (DNNs) frequently present behaviorally irregular patterns, significantly limiting their practical potentials and theoretical validity in travel behavior modeling. This study proposes strong and weak …
Domain Generalization