Learnable Embedding Space for Efficient Neural Architecture Compression
We propose a method to incrementally learn an embedding space over the domain of network architectures, to enable the careful selection of architectures for evaluation during compressed architecture search. Given a teacher network, we search for a compressed network architecture by using Bayesian Optimization (BO) with a kernel function defined over our proposed embedding space to select architectures for evaluation. We demonstrate that our search algorithm can significantly outperform various baseline methods, such as random search and reinforcement learning (Ashok et al., 2018). The compressed architectures found by our method are also better than the state-of-the-art manually-designed compact architecture ShuffleNet (Zhang et al., 2018). We also demonstrate that the learned embedding space can be transferred to new settings for architecture search, such as a larger teacher network or a teacher network in a different architecture family, without any training. Code is publicly available here: https://github.com/Friedrich1006/ESNAC .
Code (2)
Tasks
Bayesian OptimizationNeural Architecture SearchReinforcement LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Architecture Compression
In this paper we propose a novel approach to model compression termed Architecture Compression. Instead of operating on the weight or filter space of the network like classical model compression methods, our approach ope…
DecoderModel CompressionThe Future is Sparse: Embedding Compression for Scalable Retrieval in Recommender Systems
Industry-scale recommender systems face a core challenge: representing entities with high cardinality, such as users or items, using dense embeddings that must be accessible during both training and inference. However, a…
Recommendation SystemsRetrievalEfficient Implicit Neural Compression of Point Clouds via Learnable Activation in Latent Space
Implicit Neural Representations (INRs), also known as neural fields, have emerged as a powerful paradigm in deep learning, parameterizing continuous spatial fields using coordinate-based neural networks. In this paper, w…
AttributeDecoderKolmogorov-Arnold NetworksPICO+1Differentiable Product Quantization for End-to-End Embedding Compression
Embedding layers are commonly used to map discrete symbols into continuous embedding vectors that reflect their semantic meanings. Despite their effectiveness, the number of parameters in an embedding layer increases lin…
QuantizationAutoencoding-Free Context Compression for LLMs via Contextual Semantic Anchors
Context compression is an advanced technique that accelerates large language model (LLM) inference by converting long inputs into compact representations. Existing methods primarily rely on autoencoding tasks to train sp…