Towards Understanding the Importance of Shortcut Connections in Residual Networks

2019-09-10 · NeurIPS 2019 12 · Tianyi Liu, Minshuo Chen, Mo Zhou, Simon S. Du, Enlu Zhou, Tuo Zhao

Residual Network (ResNet) is undoubtedly a milestone in deep learning. ResNet is equipped with shortcut connections between layers, and exhibits efficient training using simple first order algorithms. Despite of the great empirical success, the reason behind is far from being well understood. In this paper, we study a two-layer non-overlapping convolutional ResNet. Training such a network requires solving a non-convex optimization problem with a spurious local optimum. We show, however, that gradient descent combined with proper normalization, avoids being trapped by the spurious local optimum, and converges to a global optimum in polynomial time, when the weight of the first layer is initialized at 0, and that of the second layer is initialized arbitrarily in a ball. Numerical experiments are provided to support our theory.

📄 PDF Abstract BibTeX arXiv:1909.04653

Code (0)

등록된 구현이 없습니다.

Methods 이 논문이 사용한 방법론

Average Pooling 설명 없음

ReLU How Do I Communicate to Expedia? How Do I Communicate to Expedia? – Call ☎️ +1-(888) 829 (0881) or +1-805-330-4056 or +1-805-330-4056 for Live Support & Special Travel…

1x1 Convolution A 1 x 1 Convolution is a convolution with some special properties in that it can be used for dimensionality reduction,…

Batch Normalization 설명 없음

Bottleneck Residual Block A Bottleneck Residual Block is a variant of the residual block that utilises 1x1 convolutions to create a bottleneck. The…

Global Average Pooling Global Average Pooling is a pooling operation designed to replace fully connected layers in classical CNNs. The idea is to generate one feature map for each corresponding…

Residual Block Residual Blocks are skip-connection blocks that learn residual functions with reference to the layer inputs, instead of learning unreferenced functions. They were introduced…

Kaiming Initialization 설명 없음

Visualizing Residual Networks

2017-01-09 · Brian Chu, Daylen Yang, Ravi Tadinada

Residual networks are the current state of the art on ImageNet. Similar work in the direction of utilizing shortcut connections has been done extremely recently with derivatives of residual networks and with highway netw…

Tandem Blocks in Deep Convolutional Neural Networks

2018-06-01 · ICLR 2018 1 · Chris Hettinger, Tanner Christensen, Jeffrey Humpherys, Tyler J. Jarvis

Due to the success of residual networks (resnets) and related architectures, shortcut connections have quickly become standard tools for building convolutional neural networks. The explanations in the literature for the …

General Classification image-classification Image Classification

IamNN: Iterative and Adaptive Mobile Neural Network for Efficient Image Classification

2018-04-26 · Sam Leroux, Pavlo Molchanov, Pieter Simoens, Bart Dhoedt 외

Deep residual networks (ResNets) made a recent breakthrough in deep learning. The core idea of ResNets is to have shortcut connections between layers that allow the network to be much deeper while still being easy to opt…

General Classification image-classification Image Classification

Demystifying ResNet

2016-11-03 · Sihan Li, Jiantao Jiao, Yanjun Han, Tsachy Weissman

The Residual Network (ResNet), proposed in He et al. (2015), utilized shortcut connections to significantly reduce the difficulty of training, which resulted in great performance boosts in terms of both training and gene…

ResNet: Enabling Deep Convolutional Neural Networks through Residual Learning

2025-10-28 · Xingyu Liu, Kun Ming Goh arxiv

Convolutional Neural Networks (CNNs) has revolutionized computer vision, but training very deep networks has been challenging due to the vanishing gradient problem. This paper explores Residual Networks (ResNet), introdu…