Improve Generalization Ability of Deep Wide Residual Network with A Suitable Scaling Factor

2024-03-07 · Songtao Tian, Zixiong Yu

Deep Residual Neural Networks (ResNets) have demonstrated remarkable success across a wide range of real-world applications. In this paper, we identify a suitable scaling factor (denoted by $\alpha$) on the residual branch of deep wide ResNets to achieve good generalization ability. We show that if $\alpha$ is a constant, the class of functions induced by Residual Neural Tangent Kernel (RNTK) is asymptotically not learnable, as the depth goes to infinity. We also highlight a surprising phenomenon: even if we allow $\alpha$ to decrease with increasing depth $L$, the degeneration phenomenon may still occur. However, when $\alpha$ decreases rapidly with $L$, the kernel regression with deep RNTK with early stopping can achieve the minimax rate provided that the target regression function falls in the reproducing kernel Hilbert space associated with the infinite-depth RNTK. Our simulation studies on synthetic data and real classification tasks such as MNIST, CIFAR10 and CIFAR100 support our theoretical criteria for choosing $\alpha$.

📄 PDF Abstract BibTeX arXiv:2403.04545

Code (0)

등록된 구현이 없습니다.

Tasks

regression

Methods 이 논문이 사용한 방법론

Early Stopping Early Stopping is a regularization technique for deep neural networks that stops training when parameter updates no longer begin to yield improves on a validation set. In…

Generalization Ability of Wide Residual Networks

2023-05-29 · Jianfa Lai, Zixiong Yu, Songtao Tian, Qian Lin

In this paper, we study the generalization ability of the wide residual network on $\mathbb{S}^{d-1}$ with the ReLU activation function. We first show that as the width $m\rightarrow\infty$, the residual network kernel (…

regression

Efficient Residual Learning with Mixture-of-Experts for Universal Dexterous Grasping

2024-10-03 · Ziye Huang, Haoqi Yuan, Yuhui Fu, Zongqing Lu

Universal dexterous grasping across diverse objects presents a fundamental yet formidable challenge in robot learning. Existing approaches using reinforcement learning (RL) to develop policies on extensive object dataset…

GPU Mixture-of-Experts Multi-Task Learning Reinforcement Learning (RL)

Self-Adaptive Scaling for Learnable Residual Structure

2019-11-01 · CONLL 2019 11 · Fenglin Liu, Meng Gao, Yuanxin Liu, Kai Lei

Residual has been widely applied to build deep neural networks with enhanced feature propagation and improved accuracy. In the literature, multiple variants of residual structure are proposed. However, most of them are m…

de-en Image Captioning image-classification Image Classification+2

Wide Contextual Residual Network with Active Learning for Remote Sensing Image Classification

2018-07-22 · IGARSS 2018 - 2018 IEEE International Geoscience and Remote Sensing Symposium 2018 7 · Sheng-Jie Liu, Haowen Luo, Ying Tu, Zhi He 외

In this paper, we propose a wide contextual residual network (WCRN) with active learning (AL) for remote sensing image (RSI) classification. Although ResNets have achieved great success in various applications (e.g. RSI…

Active Learning Classification General Classification Hyperspectral Image Classification+3

Unsupervised Domain Adaptation with Adversarial Residual Transform Networks

2018-04-25 · Guanyu Cai, Yuqin Wang, Mengchu Zhou, Lianghua He

Domain adaptation is widely used in learning problems lacking labels. Recent studies show that deep adversarial domain adaptation models can make markable improvements in performance, which include symmetric and asymmetr…

Domain Adaptation Unsupervised Domain Adaptation