DropFilter: A Novel Regularization Method for Learning Convolutional Neural Networks

2018-11-16 · Hengyue Pan, Hui Jiang, Xin Niu, Yong Dou

The past few years have witnessed the fast development of different regularization methods for deep learning models such as fully-connected deep neural networks (DNNs) and Convolutional Neural Networks (CNNs). Most of previous methods mainly consider to drop features from input data and hidden layers, such as Dropout, Cutout and DropBlocks. DropConnect select to drop connections between fully-connected layers. By randomly discard some features or connections, the above mentioned methods control the overfitting problem and improve the performance of neural networks. In this paper, we proposed two novel regularization methods, namely DropFilter and DropFilter-PLUS, for the learning of CNNs. Different from the previous methods, DropFilter and DropFilter-PLUS selects to modify the convolution filters. For DropFilter-PLUS, we find a suitable way to accelerate the learning process based on theoretical analysis. Experimental results on MNIST show that using DropFilter and DropFilter-PLUS may improve performance on image classification tasks.

📄 PDF Abstract BibTeX arXiv:1811.06783

Code (0)

등록된 구현이 없습니다.

Tasks

image-classification Image Classification

Methods 이 논문이 사용한 방법론

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Cutout Cutout is an image augmentation and regularization technique that randomly masks out square regions of input during training. and can be used to improve the robustness and…

DropConnect DropConnect generalizes Dropout by randomly dropping the weights rather than the activations with probability $1-p$. DropConnect…

Convolution A convolution is a type of matrix operation, consisting of a kernel, a small matrix of weights, that slides over input data performing element-wise multiplication with the…

DropFilter: Dropout for Convolutions

2018-10-23 · Zhengsu Chen Jianwei Niu Qi Tian

Using a large number of parameters , deep neural networks have achieved remarkable performance on computer vison and natural language processing tasks. However the networks usually suffer from overfitting by using too mu…

Implicit Regularization in Hierarchical Tensor Factorization and Deep Convolutional Neural Networks

2022-01-27 · Noam Razin, Asaf Maman, Nadav Cohen

In the pursuit of explaining implicit regularization in deep learning, prominent focus was given to matrix and tensor factorizations, which correspond to simplified neural networks. It was shown that these models exhibit…

Existence, Stability and Scalability of Orthogonal Convolutional Neural Networks

2021-08-12 · El Mehdi Achour, François Malgouyres, Franck Mamalet

Imposing orthogonality on the layers of neural networks is known to facilitate the learning by limiting the exploding/vanishing of the gradient; decorrelate the features; improve the robustness. This paper studies the th…

Filter Bank Regularization of Convolutional Neural Networks

2019-07-25 · Seyed Mehdi Ayyoubzadeh, Xiaolin Wu

Regularization techniques are widely used to improve the generality, robustness, and efficiency of deep convolutional neural networks (DCNNs). In this paper, we propose a novel approach of regulating DCNN convolutional k…

Object Recognition

Convolutional Neural Networks with Dynamic Regularization

2019-09-26 · Yi Wang, Zhen-Peng Bian, Junhui Hou, Lap-Pui Chau

Regularization is commonly used for alleviating overfitting in machine learning. For convolutional neural networks (CNNs), regularization methods, such as DropBlock and Shake-Shake, have illustrated the improvement in th…