Rethinking Channel Dimensions for Efficient Model Design
Designing an efficient model within the limited computational cost is challenging. We argue the accuracy of a lightweight model has been further limited by the design convention: a stage-wise configuration of the channel dimensions, which looks like a piecewise linear function of the network stage. In this paper, we study an effective channel dimension configuration towards better performance than the convention. To this end, we empirically study how to design a single layer properly by analyzing the rank of the output feature. We then investigate the channel configuration of a model by searching network architectures concerning the channel configuration under the computational cost restriction. Based on the investigation, we propose a simple yet effective channel configuration that can be parameterized by the layer index. As a result, our proposed model following the channel parameterization achieves remarkable performance on ImageNet classification and transfer learning tasks including COCO object detection, COCO instance segmentation, and fine-grained classifications. Code and ImageNet pretrained models are available at https://github.com/clovaai/rexnet.
Code (10)
Tasks
Image ClassificationInstance Segmentationmodelobject-detectionObject DetectionSemantic SegmentationTransfer LearningSimilar Papers 제목 키워드 기반
Rethinking Transformer-Based Blind-Spot Network for Self-Supervised Image Denoising
Blind-spot networks (BSN) have been prevalent neural architectures in self-supervised image denoising (SSID). However, most existing BSNs are conducted with convolution layers. Although transformers have shown the potent…
Computational EfficiencyDenoisingImage DenoisingImage Restoration+1Rethinking Spatial Dimensions of Vision Transformers
Vision Transformer (ViT) extends the application range of transformers from language processing to computer vision tasks as being an alternative architecture against the existing convolutional neural networks (CNN). Sinc…
Dimensionality Reductionimage-classificationImage Classificationobject-detection+1Rethinking Channel Dimensions to Isolate Outliers for Low-bit Weight Quantization of Large Language Models
Large Language Models (LLMs) have recently demonstrated remarkable success across various tasks. However, efficiently serving LLMs has been a challenge due to the large memory bottleneck, specifically in small batch infe…
HumanEvalLanguage ModelingLanguage ModellingMMLU+1Flexible Channel Dimensions for Differentiable Architecture Search
Finding optimal channel dimensions (i.e., the number of filters in DNN layers) is essential to design DNNs that perform well under computational resource constraints. Recent work in neural architecture search aims at aut…
GPUNeural Architecture SearchRethinking Gradient Operator for Exposing AI-enabled Face Forgeries
For image forensics, convolutional neural networks (CNNs) tend to learn content features rather than subtle manipulation traces, which limits forensic performance. Existing methods predominantly solve the above challenge…
Image Forensics