Loss Landscape Sightseeing with Multi-Point Optimization
We present multi-point optimization: an optimization technique that allows to train several models simultaneously without the need to keep the parameters of each one individually. The proposed method is used for a thorough empirical analysis of the loss landscape of neural networks. By extensive experiments on FashionMNIST and CIFAR10 datasets we demonstrate two things: 1) loss surface is surprisingly diverse and intricate in terms of landscape patterns it contains, and 2) adding batch normalization makes it more smooth. Source code to reproduce all the reported results is available on GitHub: https://github.com/universome/loss-patterns.
Code (1)
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
A Deep Neural Network's Loss Surface Contains Every Low-dimensional Pattern
The work "Loss Landscape Sightseeing with Multi-Point Optimization" (Skorokhodov and Burtsev, 2019) demonstrated that one can empirically find arbitrary 2D binary patterns inside loss surfaces of popular neural networks.…
Affective Recommendation System for Tourists by Using Emotion Generating Calculations
An emotion orientated intelligent interface consists of Emotion Generating Calculations (EGC) and Mental State Transition Network (MSTN). We have developed the Android EGC application software which the agent works to ev…
Landscape-Awareness for Geometric View Diffusion Model
Accurate camera viewpoint estimation under sparse-view conditions remains challenging, particularly in two-view scenarios. Recent approaches leverage diffusion models such as Zero123 to synthesize novel views conditioned…
Embedding Principle of Loss Landscape of Deep Neural Networks
Understanding the structure of loss landscape of deep neural networks (DNNs)is obviously important. In this work, we prove an embedding principle that the loss landscape of a DNN "contains" all the critical points of all…
Protein FoldingThe loss landscape of deep linear neural networks: a second-order analysis
We study the optimization landscape of deep linear neural networks with the square loss. It is known that, under weak assumptions, there are no spurious local minima and no local maxima. However, the existence and divers…
Diversity