On skip connections and normalisation layers in deep optimisation

2022-10-10 · NeurIPS 2023 11 · Lachlan Ewen MacDonald, Jack Valmadre, Hemanth Saratchandran, Simon Lucey

We introduce a general theoretical framework, designed for the study of gradient optimisation of deep neural networks, that encompasses ubiquitous architecture choices including batch normalisation, weight normalisation and skip connections. Our framework determines the curvature and regularity properties of multilayer loss landscapes in terms of their constituent layers, thereby elucidating the roles played by normalisation layers and skip connections in globalising these properties. We then demonstrate the utility of this framework in two respects. First, we give the only proof of which we are aware that a class of deep neural networks can be trained using gradient descent to global optima even when such optima only exist at infinity, as is the case for the cross-entropy cost. Second, we identify a novel causal mechanism by which skip connections accelerate training, which we verify predictively with ResNets on MNIST, CIFAR10, CIFAR100 and ImageNet.

📄 PDF Abstract BibTeX arXiv:2210.05371

Code (0)

등록된 구현이 없습니다.

Methods 이 논문이 사용한 방법론

AWARE We propose to theoretically and empirically examine the effect of incorporating weighting schemes into walk-aggregating GNNs. To this end, we propose a simple, interpretable, and…

Simplifying Transformer Blocks

2023-11-03 · Bobby He, Thomas Hofmann

A simple design recipe for deep Transformers is to compose identical building blocks. But standard transformer blocks are far from simple, interweaving attention and MLP sub-blocks with skip connections & normalisation l…

Decoder

Normalisation is dead, long live normalisation!

2022-01-17 · ICLR Track Blog 2022 5 · Anonymous

Since the advent of Batch Normalisation (BN) almost every state-of-the-art (SOTA) method uses some form of normalisation. After all, normalisation generally speeds up learning and leads to models that generalise better t…

Form

Deep Transformers without Shortcuts: Modifying Self-attention for Faithful Signal Propagation

2023-02-20 · Bobby He, James Martens, Guodong Zhang, Aleksandar Botev 외

Skip connections and normalisation layers form two standard architectural components that are ubiquitous for the training of Deep Neural Networks (DNNs), but whose precise roles are poorly understood. Recent approaches s…

Orthogonal Self-Attention

2026-02-05 · Leo Zhang, James Martens arxiv

Softmax Self-Attention (SSA) is a key component of Transformer architectures. However, when utilised within skipless architectures, which aim to improve representation learning, recent work has highlighted the inherent i…

Representation Learning

New Insights into Graph Convolutional Networks using Neural Tangent Kernels

2021-10-08 · Mahalakshmi Sabanayagam, Pascal Esser, Debarghya Ghoshdastidar

Graph Convolutional Networks (GCNs) have emerged as powerful tools for learning on network structured data. Although empirically successful, GCNs exhibit certain behaviour that has no rigorous explanation -- for instance…