ScaleNet: Searching for the Model to Scale
Recently, community has paid increasing attention on model scaling and contributed to developing a model family with a wide spectrum of scales. Current methods either simply resort to a one-shot NAS manner to construct a non-structural and non-scalable model family or rely on a manual yet fixed scaling strategy to scale an unnecessarily best base model. In this paper, we bridge both two components and propose ScaleNet to jointly search base model and scaling strategy so that the scaled large model can have more promising performance. Concretely, we design a super-supernet to embody models with different spectrum of sizes (e.g., FLOPs). Then, the scaling strategy can be learned interactively with the base model via a Markov chain-based evolution algorithm and generalized to develop even larger models. To obtain a decent super-supernet, we design a hierarchical sampling strategy to enhance its training sufficiency and alleviate the disturbance. Experimental results show our scaled networks enjoy significant performance superiority on various FLOPs, but with at least 2.53x reduction on search cost. Codes are available at https://github.com/luminolx/ScaleNet.
Code (1)
Tasks
modelMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
ScaleNAS: One-Shot Learning of Scale-Aware Representations for Visual Recognition
Scale variance among different sizes of body parts and objects is a challenging problem for visual recognition tasks. Existing works usually design dedicated backbone or apply Neural architecture Search(NAS) for each tas…
Multi-Person Pose EstimationNeural Architecture SearchOne-Shot LearningPose Estimation+1ScaleNet: A Shallow Architecture for Scale Estimation
In this paper, we address the problem of estimating scale factors between images. We formulate the scale estimation problem as a prediction of a probability distribution over scale factors. We design a new architecture, …
3D ReconstructionCamera Pose EstimationGeometric MatchingPose EstimationScaleNet: Scaling up Pretrained Neural Networks with Incremental Parameters
Recent advancements in vision transformers (ViTs) have demonstrated that larger models often achieve superior performance. However, training these models remains computationally intensive and costly. To address this chal…
Image ClassificationObject DetectionScaleNet: An Unsupervised Representation Learning Method for Limited Information
Although large-scale labeled data are essential for deep convolutional neural networks (ConvNets) to learn high-level semantic visual representations, it is time-consuming and impractical to collect and annotate large-sc…
Representation LearningEcoScaleNet: A Lightweight Multi Kernel Network for Long Sequence 12 lead ECG Classification
Accurate interpretation of 12 lead electrocardiograms (ECGs) is critical for early detection of cardiac abnormalities, yet manual reading is error prone and existing CNN based classifiers struggle to choose receptive fie…
ECG Classification