How Gradient Descent Separates Data with Neural Collapse: A Layer-Peeled Perspective
In this paper, we derive a landscape analysis to the surrogate model to study the inductive bias of the neural features and parameters from neural networks with cross-entropy. We show that once the training cross-entropy loss decreases below a certain threshold, the features and classifiers in the last layer of the neural network will converge to a certain geometry structure, which is known as neural collapse\citep{papyan2020prevalence,fang2021layer}, \emph{i.e.} cross-example within-class variability of last-layer feature collapses to zero and the class-means converge to a Simplex Equiangular Tight Frame (ETF). We illustrate that the cross-entropy loss enjoys a benign global landscape where all the critical points are strict saddles whose Hessian exhibit negative curvature directions except the only global minimizers which exhibit neural collapse phenomenon.
Code (0)
등록된 구현이 없습니다.
Tasks
Inductive BiasSimilar Papers 제목 키워드 기반
Imitate Optimal Policy: Prevail and Induce Action Collapse in Policy Gradient
Policy gradient (PG) methods in reinforcement learning frequently utilize deep neural networks (DNNs) to learn a shared backbone of feature representations used to compute likelihoods in an action selection layer. Numero…
Reinforcement LearningOpenAI GymDeep Neural Collapse Is Provably Optimal for the Deep Unconstrained Features Model
Neural collapse (NC) refers to the surprising structure of the last layer of deep neural networks in the terminal phase of gradient descent training. Recently, an increasing amount of experimental evidence has pointed to…
Binary ClassificationNeural Collapse versus Low-rank Bias: Is Deep Neural Collapse Really Optimal?
Deep neural networks (DNNs) exhibit a surprising structure in their final layer known as neural collapse (NC), and a growing body of works has currently investigated the propagation of neural collapse to earlier layers o…
Binary ClassificationMulti-class ClassificationWide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapse
Deep neural networks (DNNs) at convergence consistently represent the training data in the last layer via a highly symmetric geometric structure referred to as neural collapse. This empirical evidence has spurred a line …
Deriving Decoder-Free Sparse Autoencoders from First Principles
Gradient descent on log-sum-exp (LSE) objectives performs implicit expectation--maximization (EM): the gradient with respect to each component output equals its responsibility. The same theory predicts collapse without v…