Multi-level Residual Networks from Dynamical Systems View

2017-10-27 · ICLR 2018 1 · Bo Chang, Lili Meng, Eldad Haber, Frederick Tung, David Begert

Deep residual networks (ResNets) and their variants are widely used in many computer vision applications and natural language processing tasks. However, the theoretical principles for designing and training ResNets are still not fully understood. Recently, several points of view have emerged to try to interpret ResNet theoretically, such as unraveled view, unrolled iterative estimation and dynamical systems view. In this paper, we adopt the dynamical systems point of view, and analyze the lesioning properties of ResNet both theoretically and experimentally. Based on these analyses, we additionally propose a novel method for accelerating ResNet training. We apply the proposed method to train ResNets and Wide ResNets for three image classification benchmarks, reducing training time by more than 40% with superior or on-par accuracy.

📄 PDF Abstract BibTeX arXiv:1710.10348

Code (0)

등록된 구현이 없습니다.

Tasks

General Classification image-classification Image Classification

Methods 이 논문이 사용한 방법론

Average Pooling 설명 없음

ReLU How Do I Communicate to Expedia? How Do I Communicate to Expedia? – Call ☎️ +1-(888) 829 (0881) or +1-805-330-4056 or +1-805-330-4056 for Live Support & Special Travel…

1x1 Convolution A 1 x 1 Convolution is a convolution with some special properties in that it can be used for dimensionality reduction,…

Batch Normalization 설명 없음

Bottleneck Residual Block A Bottleneck Residual Block is a variant of the residual block that utilises 1x1 convolutions to create a bottleneck. The…

Global Average Pooling Global Average Pooling is a pooling operation designed to replace fully connected layers in classical CNNs. The idea is to generate one feature map for each corresponding…

Residual Block Residual Blocks are skip-connection blocks that learn residual functions with reference to the layer inputs, instead of learning unreferenced functions. They were introduced…

Kaiming Initialization 설명 없음

Training of deep residual networks with stochastic MG/OPT

2021-08-09 · Cyrill von Planta, Alena Kopanicakova, Rolf Krause

We train deep residual networks with a stochastic variant of the nonlinear multigrid method MG/OPT. To build the multilevel hierarchy, we use the dynamical systems viewpoint specific to residual networks. We report signi…

Globally Convergent Multilevel Training of Deep Residual Networks

2021-07-15 · Alena Kopaničáková, Rolf Krause

We propose a globally convergent multilevel training method for deep residual networks (ResNets). The devised method can be seen as a novel variant of the recursive multilevel trust-region (RMTR) method, which operates i…

Multilevel Minimization for Deep Residual Networks

2020-04-13 · Lisa Gaedke-Merzhäuser, Alena Kopaničáková, Rolf Krause

We present a new multilevel minimization framework for the training of deep residual networks (ResNets), which has the potential to significantly reduce training time and effort. Our framework is based on the dynamical s…

Convolutional Neural Networks combined with Runge-Kutta Methods

2018-02-24 · ICLR 2019 5 · Mai Zhu, Bo Chang, Chong Fu

A convolutional neural network can be constructed using numerical methods for solving dynamical systems, since the forward pass of the network can be regarded as a trajectory of a dynamical system. However, existing mode…

Image Classification

Predict Globally, Correct Locally: Parallel-in-Time Optimal Control of Neural Networks

2019-02-07 · Panos Parpas, Corey Muir

The links between optimal control of dynamical systems and neural networks have proved beneficial both from a theoretical and from a practical point of view. Several researchers have exploited these links to investigate …

Distributed Optimization