Eigenvalues of Autoencoders in Training and at Initialization
In this paper, we investigate the evolution of autoencoders near their initialization. In particular, we study the distribution of the eigenvalues of the Jacobian matrices of autoencoders early in the training process, training on the MNIST data set. We find that autoencoders that have not been trained have eigenvalue distributions that are qualitatively different from those which have been trained for a long time ($>$100 epochs). Additionally, we find that even at early epochs, these eigenvalue distributions rapidly become qualitatively similar to those of the fully trained autoencoders. We also compare the eigenvalues at initialization to pertinent theoretical work on the eigenvalues of random matrices and the products of such matrices.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Autocorrelation Matters: Understanding the Role of Initialization Schemes for State Space Models
Current methods for initializing state space model (SSM) parameters primarily rely on the HiPPO framework \citep{gu2023how}, which is based on online function approximation with the SSM kernel basis. However, the HiPPO f…
State Space ModelsIKUN: Initialization to Keep snn training and generalization great with sUrrogate-stable variaNce
Weight initialization significantly impacts the convergence and performance of neural networks. While traditional methods like Xavier and Kaiming initialization are widely used, they often fall short for spiking neural n…
The Goldilocks zone: Towards better understanding of neural network loss landscapes
We explore the loss landscape of fully-connected and convolutional neural networks using random, low-dimensional hyperplanes and hyperspheres. Evaluating the Hessian, $H$, of the loss function on these hypersurfaces, we …
Implicit Greedy Rank Learning in Autoencoders via Overparameterized Linear Networks
Deep linear networks trained with gradient descent yield low rank solutions, as is typically studied in matrix factorization. In this paper, we take a step further and analyze implicit rank regularization in autoencoders…
SensitivityOn compression rate of quantum autoencoders: Control design, numerical and experimental realization
Quantum autoencoders which aim at compressing quantum information in a low-dimensional latent space lie in the heart of automatic data compression in the field of quantum information. In this paper, we establish an upper…
Data Compression