ELSA: Partial Weight Freezing for Overhead-Free Sparse Network Deployment
We present ELSA, a practical solution for creating deep networks that can easily be deployed at different levels of sparsity. The core idea is to embed one or more sparse networks within a single dense network as a proper subset of the weights. At prediction time, any sparse model can be extracted effortlessly simply be zeroing out weights according to a predefined mask. ELSA is simple, powerful and highly flexible. It can use essentially any existing technique for network sparsification and network training. In particular, it does not restrict the loss function, architecture or the optimization technique. Our experiments show that ELSA's advantages of flexible deployment comes with no or just a negligible reduction in prediction quality compared to the standard way of using multiple sparse networks that are trained and stored independently.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Layerwise Progressive Freezing Enables STE-Free Training of Deep Binary Neural Networks
We investigate progressive freezing as an alternative to straight-through estimators (STE) for training binary networks from scratch. Under controlled training conditions, we find that while global progressive freezing w…
Weight Freezing: A Regularization Approach for Fully Connected Layers with an Application in EEG Classification
In the realm of EEG decoding, enhancing the performance of artificial neural networks (ANNs) carries significant potential. This study introduces a novel approach, termed "weight freezing", that is anchored on the princi…
Decision MakingEEGEeg DecodingLayer Freezing & Data Sieving: Missing Pieces of a Generic Framework for Sparse Training
Recently, sparse training has emerged as a promising paradigm for efficient deep learning on edge devices. The current research mainly devotes efforts to reducing training costs by further increasing model sparsity. Howe…
Energy and Memory-Efficient Federated Learning With Ordered Layer Freezing
Federated Learning (FL) has emerged as a privacy-preserving paradigm for training machine learning models across distributed edge devices in the Internet of Things (IoT). By keeping data local and coordinating model trai…
Federated LearningRethinking the Potential of Layer Freezing for Efficient DNN Training
With the growing size of deep neural networks and datasets, the computational costs of training have significantly increased. The layer-freezing technique has recently attracted great attention as a promising method to e…