Tensor Programs IIb: Architectural Universality of Neural Tangent Kernel Training Dynamics

2021-05-08 · Greg Yang, Etai Littwin

Yang (2020a) recently showed that the Neural Tangent Kernel (NTK) at initialization has an infinite-width limit for a large class of architectures including modern staples such as ResNet and Transformers. However, their analysis does not apply to training. Here, we show the same neural networks (in the so-called NTK parametrization) during training follow a kernel gradient descent dynamics in function space, where the kernel is the infinite-width NTK. This completes the proof of the *architectural universality* of NTK behavior. To achieve this result, we apply the Tensor Programs technique: Write the entire SGD dynamics inside a Tensor Program and analyze it via the Master Theorem. To facilitate this proof, we develop a graphical notation for Tensor Programs.

📄 PDF Abstract BibTeX arXiv:2105.03703

Code (0)

등록된 구현이 없습니다.

Methods 이 논문이 사용한 방법론

NTK 설명 없음

1x1 Convolution A 1 x 1 Convolution is a convolution with some special properties in that it can be used for dimensionality reduction,…

Average Pooling 설명 없음

Batch Normalization 설명 없음

Residual Connection 설명 없음

Kaiming Initialization 설명 없음

Residual Block Residual Blocks are skip-connection blocks that learn residual functions with reference to the layer inputs, instead of learning unreferenced functions. They were introduced…

Global Average Pooling Global Average Pooling is a pooling operation designed to replace fully connected layers in classical CNNs. The idea is to generate one feature map for each corresponding…

Tensor Programs IVb: Adaptive Optimization in the Infinite-Width Limit

2023-08-03 · Greg Yang, Etai Littwin

Going beyond stochastic gradient descent (SGD), what new phenomena emerge in wide neural networks trained by adaptive optimizers like Adam? Here we show: The same dichotomy between feature learning and kernel behaviors (…

Tensor Programs II: Neural Tangent Kernel for Any Architecture

2020-06-25 · Greg Yang

We prove that a randomly initialized neural network of *any architecture* has its Tangent Kernel (NTK) converge to a deterministic limit, as the network widths tend to infinity. We demonstrate how to calculate this limit…

Feature Learning in Infinite-Width Neural Networks

2020-11-30 · Greg Yang, Edward J. Hu

As its width tends to infinity, a deep neural network's behavior under gradient descent can become simplified and predictable (e.g. given by the Neural Tangent Kernel (NTK)), if it is parametrized appropriately (e.g. the…

Few-Shot Learning Transfer Learning

A Kernel-Based View of Language Model Fine-Tuning

2022-10-11 · Sadhika Malladi, Alexander Wettig, Dingli Yu, Danqi Chen 외

It has become standard to solve NLP tasks by fine-tuning pre-trained language models (LMs), especially in low-data settings. There is minimal theoretical understanding of empirical success, e.g., why fine-tuning a model …

Language Modeling Language Modelling

Optimal Kernel Orchestration for Tensor Programs with Korch

2024-06-13 · Muyan Hu, Ashwin Venkatram, Shreyashri Biswas, Balamurugan Marimuthu 외

Kernel orchestration is the task of mapping the computation defined in different operators of a deep neural network (DNN) to the execution of GPU kernels on modern hardware platforms. Prior approaches optimize kernel orc…

Diversity GPU tensor algebra