paper-with-me

홈 › Papers

Linear Stability Hypothesis and Rank Stratification for Nonlinear Models

2022-11-21 · Yaoyu Zhang, Zhongwang Zhang, Leyang Zhang, Zhiwei Bai, Tao Luo, Zhi-Qin John Xu

Models with nonlinear architectures/parameterizations such as deep neural networks (DNNs) are well known for their mysteriously good generalization performance at overparameterization. In this work, we tackle this mystery from a novel perspective focusing on the transition of the target recovery/fitting accuracy as a function of the training data size. We propose a rank stratification for general nonlinear models to uncover a model rank as an "effective size of parameters" for each function in the function space of the corresponding model. Moreover, we establish a linear stability theory proving that a target function almost surely becomes linearly stable when the training data size equals its model rank. Supported by our experiments, we propose a linear stability hypothesis that linearly stable functions are preferred by nonlinear training. By these results, model rank of a target function predicts a minimal training data size for its successful recovery. Specifically for the matrix factorization model and DNNs of fully-connected or convolutional architectures, our rank stratification shows that the model rank for specific target functions can be much lower than the size of model parameters. This result predicts the target recovery capability even at heavy overparameterization for these nonlinear models as demonstrated quantitatively by our experiments. Overall, our work provides a unified framework with quantitative prediction power to understand the mysterious target recovery behavior at overparameterization for general nonlinear models.

📄 PDF Abstract BibTeX arXiv:2211.11623

Code (0)

등록된 구현이 없습니다.

Similar Papers 제목 키워드 기반

Learning-Based Stable Optimal Control for Infinite-Time Nonlinear Regulation Problems

2025-06-12 · Han Wang, Di wu, Lin Cheng, Shengping Gong 외

Infinite-time nonlinear optimal regulation control is widely utilized in aerospace engineering as a systematic method for synthesizing stable controllers. However, conventional methods often rely on linearization hypothe…

Neural Identification for Control

2020-09-24 · Priyabrata Saha, Magnus Egerstedt, Saibal Mukhopadhyay

We present a new method for learning control law that stabilizes an unknown nonlinear dynamical system at an equilibrium point. We formulate a system identification task in a self-supervised learning setting that jointly…

Self-Supervised Learning

Variance Reduction for Heavy-Tailed Monetization Metrics in Ranking Experiments via Post-Stratification

2026-06-02 · Neeti Pokharna, Olivier Jeunen, Yatharth Saraf, Aleksei Ustimenko arxiv

Online evaluation of ranking and retrieval systems often relies on downstream monetization metrics such as app revenue or creator earnings. These metrics are typically heavy-tailed, with a small fraction of users dominat…

Recommendation SystemsInformation Retrieval

Nonlinear Adaptive Algorithms on Rank-One Tensor Models

2016-10-24 · Felipe C. Pinheiro, Cassio G. Lopes

This work proposes a low complexity nonlinearity model and develops adaptive algorithms over it. The model is based on the decomposable---or rank-one, in tensor language---Volterra kernels. It may also be described as a …

Stability Verification in Stochastic Control Systems via Neural Network Supermartingales

2021-12-17 · Mathias Lechner, Đorđe Žikelić, Krishnendu Chatterjee, Thomas A. Henzinger

We consider the problem of formally verifying almost-sure (a.s.) asymptotic stability in discrete-time nonlinear stochastic control systems. While verifying stability in deterministic control systems is extensively studi…