Mixed-precision Supernet Training from Vision Foundation Models using Low Rank Adapter
Compression of large and performant vision foundation models (VFMs) into arbitrary bit-wise operations (BitOPs) allows their deployment on various hardware. We propose to fine-tune a VFM to a mixed-precision quantized supernet. The supernet-based neural architecture search (NAS) can be adopted for this purpose, which trains a supernet, and then subnets within arbitrary hardware budgets can be extracted. However, existing methods face difficulties in optimizing the mixed-precision search space and incurring large memory costs during training. To tackle these challenges, first, we study the effective search space design for fine-tuning a VFM by comparing different operators (such as resolution, feature size, width, depth, and bit-widths) in terms of performance and BitOPs reduction. Second, we propose memory-efficient supernet training using a low-rank adapter (LoRA) and a progressive training strategy. The proposed method is evaluated for the recently proposed VFM, Segment Anything Model, fine-tuned on segmentation tasks. The searched model yields about a 95% reduction in BitOPs without incurring performance degradation.
Code (0)
등록된 구현이 없습니다.
Tasks
Neural Architecture SearchMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Quasar-ViT: Hardware-Oriented Quantization-Aware Architecture Search for Vision Transformers
Vision transformers (ViTs) have demonstrated their superior accuracy for computer vision tasks compared to convolutional neural networks (CNNs). However, ViT models are often computation-intensive for efficient deploymen…
QuantizationBatchQuant: Quantized-for-all Architecture Search with Robust Quantizer
As the applications of deep learning models on edge devices increase at an accelerating pace, fast adaptation to various scenarios with varying resource constraints has become a crucial aspect of model deployment. As a r…
AllGPUHardware Aware Neural Architecture SearchModel Optimization+2Neural Architecture Search as Sparse Supernet
This paper aims at enlarging the problem of Neural Architecture Search (NAS) from Single-Path and Multi-Path Search to automated Mixed-Path Search. In particular, we model the NAS problem as a sparse supernet using a new…
Neural Architecture SearchEffective and Fast: A Novel Sequential Single Path Search for Mixed-Precision Quantization
Since model quantization helps to reduce the model size and computation latency, it has been successfully applied in many applications of mobile phones, embedded devices and smart chips. The mixed-precision quantization …
QuantizationOne-Shot Model for Mixed-Precision Quantization
Neural network quantization is a popular approach for model compression. Modern hardware supports quantization in mixed-precision mode, which allows for greater compression rates but adds the challenging task of sear…
modelModel CompressionQuantizationSuper-Resolution