STARFISH: faST Accuracy Recovery in pruned networks From Internal State Healing
Pruning is a process designed to reduce the number of weights in a large neural network. This can substantially speed up inference but might cause a considerable reduction in the model's accuracy, and thus it is usually followed by a healing process that regains some of the lost accuracy. In this paper, we propose a new healing method, STARFISH, that can recover (most of) the accuracy of any pruned network efficiently. The main idea of STARFISH is to optimize the pruned network to align with the original network's internal state representations using a tiny calibration set of unlabeled examples. For the common case of removing 50% of the weights, STARFISH healing improves the recovered accuracy by up to 22% over the state-of-the-art methods on ViT-based networks. Its advantage is even more pronounced under aggressive pruning. For example, after eliminating 75% of the weights in a DeiT-B network for ImageNet, STARFISH uses only 0.4% of the number of training images as a calibration set and recovers 82% of the original dense accuracy, whereas competing recovery techniques reach only 40% of the dense model accuracy.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Distilled Gradual Pruning with Pruned Fine-tuning
Neural Networks (NNs) have been driving machine learning progress in recent years, but their larger models present challenges in resource-limited environments. Weight pruning reduces the computational demand, often with …
Image ClassificationKnowledge DistillationSelf-Knowledge DistillationStarfish: A Prototype for Universal Preprocessing and Text-Embedded Programming
We present a novel concept of universal text preprocessing and text-embedded programming (PTEP). Preprocessing and text-embedded programming has been widely used in programming languages and frameworks in a fragmented an…
Cyclical Pruning for Sparse Neural Networks
Current methods for pruning neural network weights iteratively apply magnitude-based pruning on the model weights and re-train the resulting model to recover lost accuracy. In this work, we show that such strategies do n…
Restoring Pruned Large Language Models via Lost Component Compensation
Pruning is a widely used technique to reduce the size and inference cost of large language models (LLMs), but it often causes performance degradation. To mitigate this, existing restoration methods typically employ param…
parameter-efficient fine-tuningThe CSIRO Crown-of-Thorn Starfish Detection Dataset
Crown-of-Thorn Starfish (COTS) outbreaks are a major cause of coral loss on the Great Barrier Reef (GBR) and substantial surveillance and control programs are underway in an attempt to manage COTS populations to ecologic…
BIG-bench Machine LearningManagement