GradMax: Growing Neural Networks using Gradient Information
The architecture and the parameters of neural networks are often optimized independently, which requires costly retraining of the parameters whenever the architecture is modified. In this work we instead focus on growing the architecture without requiring costly retraining. We present a method that adds new neurons during training without impacting what is already learned, while improving the training dynamics. We achieve the latter by maximizing the gradients of the new weights and find the optimal initialization efficiently by means of the singular value decomposition (SVD). We call this technique Gradient Maximizing Growth (GradMax) and demonstrate its effectiveness in variety of vision tasks and architectures.
Code (1)
Similar Papers 제목 키워드 기반
Differentiable Quality Diversity
Quality diversity (QD) is a growing branch of stochastic optimization research that studies the problem of generating an archive of solutions that maximize a given objective function but are also diverse with respect to …
DiversityStochastic OptimizationBayesian Uncertainty for Gradient Aggregation in Multi-Task Learning
As machine learning becomes more prominent there is a growing demand to perform several inference tasks in parallel. Running a dedicated model for each task is computationally expensive and therefore there is a great int…
Bayesian InferenceMulti-Task LearningMomentum Gradient Descent Federated Learning with Local Differential Privacy
Nowadays, the development of information technology is growing rapidly. In the big data era, the privacy of personal information has been more pronounced. The major challenge is to find a way to guarantee that sensitive …
Federated LearningPrivacy PreservingBuilding Gradient Bridges: Label Leakage from Restricted Gradient Sharing in Federated Learning
The growing concern over data privacy, the benefits of utilizing data from diverse sources for model training, and the proliferation of networked devices with enhanced computational capabilities have all contributed to t…
Federated LearningQuantum Natural Policy Gradients: Towards Sample-Efficient Reinforcement Learning
Reinforcement learning is a growing field in AI with a lot of potential. Intelligent behavior is learned automatically through trial and error in interaction with the environment. However, this learning process is often …
Multi-Armed Banditsreinforcement-learningReinforcement Learning