A Frobenius norm regularization method for convolutional kernels to avoid unstable gradient problem
Convolutional neural network is a very important model of deep learning. It can help avoid the exploding/vanishing gradient problem and improve the generalizability of a neural network if the singular values of the Jacobian of a layer are bounded around $1$ in the training process. We propose a new penalty function for a convolutional kernel to let the singular values of the corresponding transformation matrix are bounded around $1$. We show how to carry out the gradient type methods. The penalty is about the structured transformation matrix corresponding to a convolutional kernel. This provides a new regularization method about the weights of convolutional layers.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
On regularization for a convolutional kernel in neural networks
Convolutional neural network is an important model in deep learning. To avoid exploding/vanishing gradient problems and to improve the generalizability of a neural network, it is desirable to have a convolution operation…
Deep Learning with Kernels through RKHM and the Perron-Frobenius Operator
Reproducing kernel Hilbert $C^*$-module (RKHM) is a generalization of reproducing kernel Hilbert space (RKHS) by means of $C^*$-algebra, and the Perron-Frobenius operator is a linear operator related to the composition o…
Deep LearningFroSSL: Frobenius Norm Minimization for Efficient Multiview Self-Supervised Learning
Self-supervised learning (SSL) is a popular paradigm for representation learning. Recent multiview methods can be classified as sample-contrastive, dimension-contrastive, or asymmetric network-based, with each family hav…
Representation LearningSelf-Supervised LearningNuclear Norm Regularization for Deep Learning
Penalizing the nuclear norm of a function's Jacobian encourages it to locally behave like a low-rank linear map. Such functions vary locally along only a handful of directions, making the Jacobian nuclear norm a natural …
Deep LearningDenoisingRepresentation LearningQuaternion Nuclear Norm minus Frobenius Norm Minimization for color image reconstruction
Color image restoration methods typically represent images as vectors in Euclidean space or combinations of three monochrome channels. However, they often overlook the correlation between these channels, leading to color…
DeblurringDenoisingImage ReconstructionImage Restoration