Essentially No Barriers in Neural Network Energy Landscape
Training neural networks involves finding minima of a high-dimensional non-convex loss function. Knowledge of the structure of this energy landscape is sparse. Relaxing from linear interpolations, we construct continuous paths between minima of recent neural network architectures on CIFAR10 and CIFAR100. Surprisingly, the paths are essentially flat in both the training and test landscapes. This implies that neural networks have enough capacity for structural changes, or that these changes are small between minima. Also, each minimum has at least one vanishing Hessian eigenvalue in addition to those resulting from trivial invariance.
Code (2)
Similar Papers 제목 키워드 기반
First-passage times in complex energy landscapes: a case study with nonmuscle myosin II assembly
Complex energy landscapes often arise in biological systems, e.g. for protein folding, biochemical reactions or intracellular transport processes. Their physical effects are often reflected in the first-passage times ari…
Protein FoldingExploring the underlying mechanisms of Xenopus laevis embryonic cell cycle
Cell cycle is an indispensable process in the proliferation and development. Despite significant efforts, global quantification and physical understanding are still challenging. In this study, we explored the mechanisms …
Drug DiscoveryNutritionEnergy landscape analysis of cardiac fibrillation wave dynamics using pairwise maximum entropy model
Cardiac fibrillation is characterized by chaotic and disintegrated spiral wave dynamics patterns, whereas sinus rhythm shows synchronized excitation patterns. To determine functional correlations among cardiomyocytes dur…
RhythmConsensus-based adaptive sampling and approximation for high-dimensional energy landscapes
We present a consensus-based framework that unifies phase space exploration with posterior-residual-based adaptive sampling for surrogate construction in high-dimensional energy landscapes. Unlike standard approximation …
Efficient ExplorationSimultaneous linear connectivity of neural networks modulo permutation
Neural networks typically exhibit permutation symmetries which contribute to the non-convexity of the networks' loss landscapes, since linearly interpolating between two permuted versions of a trained network tends to en…