Loss Landscapes of Regularized Linear Autoencoders
Autoencoders are a deep learning model for representation learning. When trained to minimize the distance between the data and its reconstruction, linear autoencoders (LAEs) learn the subspace spanned by the top principal directions but cannot learn the principal directions themselves. In this paper, we prove that $L_2$-regularized LAEs are symmetric at all critical points and learn the principal directions as the left singular vectors of the decoder. We smoothly parameterize the critical manifold and relate the minima to the MAP estimate of probabilistic PCA. We illustrate these results empirically and consider implications for PCA algorithms, computational neuroscience, and the algebraic topology of learning.
Code (2)
Tasks
DecoderRepresentation LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Non-linear, Sparse Dimensionality Reduction via Path Lasso Penalized Autoencoders
High-dimensional data sets are often analyzed and explored via the construction of a latent low-dimensional space which enables convenient visualization and efficient predictive modeling or clustering. For complex data s…
ClusteringDimensionality ReductionFeature-metric Loss for Self-supervised Learning of Depth and Egomotion
Photometric loss is widely used for self-supervised depth and egomotion estimation. However, the loss landscapes induced by photometric differences are often problematic for optimization, caused by plateau landscapes for…
Depth EstimationMonocular Depth EstimationSelf-Supervised LearningVisual OdometryNeural network optimization strategies and the topography of the loss landscape
Neural networks are trained by optimizing multi-dimensional sets of fitting parameters on non-convex loss landscapes. Low-loss regions of the landscapes correspond to the parameter sets that perform well on the training …
What can linear interpolation of neural network loss landscapes tell us?
Studying neural network loss landscapes provides insights into the nature of the underlying optimization problems. Unfortunately, loss landscapes are notoriously difficult to visualize in a human-comprehensible fashion. …
Error Bound Analysis for the Regularized Loss of Deep Linear Neural Networks
The optimization foundations of deep linear networks have received significant attention lately. However, due to the non-convexity and hierarchical structure, analyzing the regularized loss of deep linear networks remain…