Diversity Based Edge Pruning of Neural Networks Using Determinantal Point Processes
Deep learning architectures with huge number of parameters are often compressed using pruning techniques. Two classes of pruning techniques are node pruning and edge pruning. A fairly recent work established that Determinantal Point Process (DPP) based node pruning empirically outperforms competing node pruning methods. However, one prominent appeal of edge pruning over node pruning is the consistent finding in literature is that sparse neural networks (edge pruned) generalize better than dense neural networks (node pruned). Building on these previous work and drawing motivation from synaptic diversity in the brain, we propose a novel diversity-based edge pruning technique for neural networks using DPP. We then empirically show that DPP edge pruning for neural networks outperforms other competing methods (both edge and node) on real data.
Code (0)
등록된 구현이 없습니다.
Tasks
DiversityPoint ProcessesMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Diversity Networks: Neural Network Compression Using Determinantal Point Processes
We introduce Divnet, a flexible technique for learning networks with diverse neurons. Divnet models neuronal diversity by placing a Determinantal Point Process (DPP) over neurons in a given layer. It uses this DPP to sel…
DiversityNeural Network CompressionPoint ProcessesA New Many-Objective Evolutionary Algorithm Based on Determinantal Point Processes
To handle different types of Many-Objective Optimization Problems (MaOPs), Many-Objective Evolutionary Algorithms (MaOEAs) need to simultaneously maintain convergence and population diversity in the high-dimensional obje…
DiversityEvolutionary AlgorithmsPoint Processes3D-Prover: Diversity Driven Theorem Proving With Determinantal Point Processes
A key challenge in automated formal reasoning is the intractable search space, which grows exponentially with the depth of the proof. This branching is caused by the large number of candidate proof tactics which can be a…
Automated Theorem ProvingDiversityPoint ProcessesvalidReliability-Aware Determinantal Point Processes for Robust Informative Data Selection in Large Language Models
Informative data selection is a key requirement for large language models (LLMs) to minimize the amount of data required for fine-tuning, network distillation, and token pruning, enabling fast and efficient deployment, e…
Point ProcessesUnderstanding Diversity Based Neural Network Pruning in Teacher Student Setup
Despite multitude of empirical advances, there is a lack of theoretical understanding of the effectiveness of different pruning methods. We inspect different pruning techniques under the statistical mechanics formulation…
DiversityNetwork Pruning