On Tighter Generalization Bound for Deep Neural Networks: CNNs, ResNets, and Beyond
We establish a margin based data dependent generalization error bound for a general family of deep neural networks in terms of the depth and width, as well as the Jacobian of the networks. Through introducing a new characterization of the Lipschitz properties of neural network family, we achieve significantly tighter generalization bounds than existing results. Moreover, we show that the generalization bound can be further improved for bounded losses. Aside from the general feedforward deep neural networks, our results can be applied to derive new bounds for popular architectures, including convolutional neural networks (CNNs) and residual networks (ResNets). When achieving same generalization errors with previous arts, our bounds allow for the choice of larger parameter spaces of weight matrices, inducing potentially stronger expressive ability for neural networks. Numerical evaluation is also provided to support our theory.
Code (0)
등록된 구현이 없습니다.
Tasks
Generalization BoundsSimilar Papers 제목 키워드 기반
On Tighter Generalization Bounds for Deep Neural Networks: CNNs, ResNets, and Beyond
We propose a generalization error bound for a general family of deep neural networks based on the depth and width of the networks, as well as the spectral norm of weight matrices. Through introducing a novel characteriza…
Generalization BoundsGeneralization Bounds for Convolutional Neural Networks
Convolutional neural networks (CNNs) have achieved breakthrough performances in a wide range of applications including image classification, semantic segmentation, and object detection. Previous research on characterizin…
Generalization Boundsimage-classificationImage Classificationobject-detection+2Do ideas have shape? Idea registration as the continuous limit of artificial neural networks
We introduce a GP generalization of ResNets (including ResNets as a particular case). We show that ResNets (and their GP generalization) converge, in the infinite depth limit, to a generalization of image registration va…
AnatomyGaussian ProcessesImage RegistrationRBCN: Rectified Binary Convolutional Networks for Enhancing the Performance of 1-bit DCNNs
Binarized convolutional neural networks (BCNNs) are widely used to improve memory and computation efficiency of deep convolutional neural networks (DCNNs) for mobile and AI chips based applications. However, current BCNN…
BinarizationObject TrackingLearning Spatio-Temporal Features with 3D Residual Networks for Action Recognition
Convolutional neural networks with spatio-temporal 3D kernels (3D CNNs) have an ability to directly extract spatio-temporal features from videos for action recognition. Although the 3D kernels tend to overfit because of …
Action RecognitionHand-Gesture RecognitionTemporal Action Localization