Plateaus, Optima, and Overfitting in Multi-Layer Perceptrons: A Saddle-Saddle-Attractor Scenario
Vanishing gradients and overfitting are central problems in machine learning, yet are typically analyzed in asymptotic regimes that obscure their dynamical origins. Here we provide a dynamical description of learning in multi-layer perceptrons (MLPs) via a minimal model inspired by Fukumizu and Amari. We show that training dynamics traverse plateau and near-optimal regions, both organized by saddle structures, before converging to an overfitting regime. Under suitable conditions on the data, this regime collapses to a single attractor modulo symmetry. Furthermore, for finite noisy datasets, convergence to the theoretical optimum is impossible, and the dynamics necessarily settle into an overfitting solution.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Graph Contrastive Learning versus Untrained Baselines: The Role of Dataset Size
Graph Contrastive Learning (GCL) has emerged as a leading paradigm for self-supervised learning on graphs, with strong performance reported on standardized datasets and growing applications ranging from genomics to drug …
Self-Supervised LearningContrastive LearningDrug DiscoverySyntNN at SemEval-2018 Task 2: is Syntax Useful for Emoji Prediction? Embedding Syntactic Trees in Multi Layer Perceptrons
In this paper, we present SyntNN as a way to include traditional syntactic models in multilayer neural networks used in the task of Semeval Task 2 of emoji prediction. The model builds on the distributed tree embedder al…
Task 2RandomForestMLP: An Ensemble-Based Multi-Layer Perceptron Against Curse of Dimensionality
We present a novel and practical deep learning pipeline termed RandomForestMLP. This core trainable classification engine consists of a convolutional neural network backbone followed by an ensemble-based multi-layer perc…
ClassificationGeneral ClassificationAvoiding overfitting of multilayer perceptrons by training derivatives
Resistance to overfitting is observed for neural networks trained with extended backpropagation algorithm. In addition to target values, its cost function uses derivatives of those up to the $4^{\mathrm{th}}$ order. For …
Dropout Regularization in Hierarchical Mixture of Experts
Dropout is a very effective method in preventing overfitting and has become the go-to regularizer for multi-layer neural networks in recent years. Hierarchical mixture of experts is a hierarchically gated model that defi…
Mixture-of-Experts