Adaptive Low-Rank Regularization with Damping Sequences to Restrict Lazy Weights in Deep Networks
Overfitting is one of the critical problems in deep neural networks. Many regularization schemes try to prevent overfitting blindly. However, they decrease the convergence speed of training algorithms. Adaptive regularization schemes can solve overfitting more intelligently. They usually do not affect the entire network weights. This paper detects a subset of the weighting layers that cause overfitting. The overfitting recognizes by matrix and tensor condition numbers. An adaptive regularization scheme entitled Adaptive Low-Rank (ALR) is proposed that converges a subset of the weighting layers to their Low-Rank Factorization (LRF). It happens by minimizing a new Tikhonov-based loss function. ALR also encourages lazy weights to contribute to the regularization when epochs grow up. It uses a damping sequence to increment layer selection likelihood in the last generations. Thus before falling the training accuracy, ALR reduces the lazy weights and regularizes the network substantially. The experimental results show that ALR regularizes the deep networks well with high training speed and low resource usage.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Low-Complexity Models for Acoustic Scene Classification Based on Receptive Field Regularization and Frequency Damping
Deep Neural Networks are known to be very demanding in terms of computing and memory requirements. Due to the ever increasing use of embedded systems and mobile devices with a limited resource budget, designing low-compl…
Acoustic Scene ClassificationScene ClassificationRandom Subspace Cubic-Regularization Methods, with Applications to Low-Rank Functions
We propose and analyze random subspace variants of the second-order Adaptive Regularization using Cubics (ARC) algorithm. These methods iteratively restrict the search space to some random subspace of the parameters, con…
ARCEnergy-optimal control of adaptive structures
Adaptive structures are equipped with sensors and actuators to actively counteract external loads such as wind. This can significantly reduce resource consumption and emissions during the life cycle compared to conventio…
Differential Spectral Damping Gap Adaptive Regularization for Ill-Conditioned Kernel Methods
Kernel methods requiring matrix inversion -- particularly Least-Squares Twin Support Vector Machines (LSTSVM) -- suffer from exponential eigenvalue decay in their system matrices, producing severely ill-conditioned probl…
Image ReconstructionBilinear classifiers for visual recognition
We describe an algorithm for learning bilinear SVMs. Bilinear classifiers are a discriminative variant of bilinear models, which capture the dependence of data on multiple factors. Such models are particularly appropriat…
Action ClassificationGeneral ClassificationTransfer Learning