Sparsifying networks by traversing Geodesics
The geometry of weight spaces and functional manifolds of neural networks play an important role towards 'understanding' the intricacies of ML. In this paper, we attempt to solve certain open questions in ML, by viewing them through the lens of geometry, ultimately relating it to the discovery of points or paths of equivalent function in these spaces. We propose a mathematical framework to evaluate geodesics in the functional space, to find high-performance paths from a dense network to its sparser counterpart. Our results are obtained on VGG-11 trained on CIFAR-10 and MLP's trained on MNIST. Broadly, we demonstrate that the framework is general, and can be applied to a wide variety of problems, ranging from sparsification to alleviating catastrophic forgetting.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Geodesic Length Distribution in Sparse Network Ensembles
A key task in the study of networked systems is to derive local and global properties that impact connectivity, synchronizability, and robustness; computing shortest paths or geodesics yields measures of network connecti…
Traversing Geodesics to Grow Biological Structures
Biological tissues reliably grow into precise, functional structures from simple starting states during development. Throughout the developmental process, the energy of a tissue changes depending on its natural resistanc…
Encoded Prior Sliced Wasserstein AutoEncoder for learning latent manifold representations
While variational autoencoders have been successful in several tasks, the use of conventional priors are limited in their ability to encode the underlying structure of input data. We introduce an Encoded Prior Sliced Was…
VTAE: Variational Transformer Autoencoder with Manifolds Learning
Deep generative models have demonstrated successful applications in learning non-linear data distributions through a number of latent variables and these models use a nonlinear function (generator) to map latent samples …
Representation LearningGeodesics in the Deep Linear Network
We derive a general system of ODEs and associated explicit solutions in a special case for geodesics between full rank matrices in the deep linear network geometry. In the process, we find horizontal straight lines in th…