Gradient descent in higher codimension
We consider the behavior of gradient flow and of discrete and noisy gradient descent. It is commonly noted that the addition of noise to the process of discrete gradient descent can affect the trajectory of gradient descent. In previous work, we observed such effects. There, we considered the case where the minima had codimension 1. In this note, we do some computer experiments and observe the behavior of noisy gradient descent in the more complex setting of minima of higher codimension.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Implicit Bias of Projected Subgradient Method Gives Provable Robust Recovery of Subspaces of Unknown Codimension
Robust subspace recovery (RSR) is a fundamental problem in robust representation learning. Here we focus on a recently proposed RSR method termed Dual Principal Component Pursuit (DPCP) approach, which aims to recover a …
Representation LearningCenter-Manifold Reduction of Learning at Bifurcations: Interference and Rich Learning in Recurrent Neural Networks
Rich learning in recurrent neural networks often proceeds through sudden transitions in latent dynamics, but there is little theory predicting how gradient descent behaves during these events. We study the local learning…
Implicit Bias of Projected Subgradient Method Gives Provable Robust Recovery of Subspaces of Unknown Codimension
Robust subspace recovery (RSR) is a fundamental problem in robust representation learning. Although RSR has received considerable attention in the literature, there are still several aspects of RSR that remain highly un…
Representation LearningThe Global Structure of Codimension-2 Local Bifurcations in Continuous-Time Recurrent Neural Networks
If we are ever to move beyond the study of isolated special cases in theoretical neuroscience, we need to develop more general theories of neural circuits over a given neural model. The present paper considers this chall…
Emergence of Frontier Superposition: Möbius attractor and Cascade Supervision
Superposition allows Transformers to reason in depth, carrying an entire reasoning frontier in parallel through a bounded-depth forward pass instead of unrolling serial chain-of-thought tokens. While Zhu et al. (2025) ha…