Large Deviation Analysis of Function Sensitivity in Random Deep Neural Networks
Mean field theory has been successfully used to analyze deep neural networks (DNN) in the infinite size limit. Given the finite size of realistic DNN, we utilize the large deviation theory and path integral analysis to study the deviation of functions represented by DNN from their typical mean field solutions. The parameter perturbations investigated include weight sparsification (dilution) and binarization, which are commonly used in model simplification, for both ReLU and sign activation functions. We find that random networks with ReLU activation are more robust to parameter perturbations with respect to their counterparts with sign activation, which arguably is reflected in the simplicity of the functions they generate.
Code (0)
등록된 구현이 없습니다.
Tasks
BinarizationSensitivityMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Graphon-based sensitivity analysis of SIS epidemics
In this work, we use the spectral properties of graphons to study stability and sensitivity to noise of deterministic SIS epidemics over large networks. We consider the presence of additive noise in a linearized SIS mode…
SensitivityIdentifying the Frontier Structural Function and Bounding Mean Deviations
This paper analyzes a model in which an outcome variable equals a frontier function of inputs less a nonnegative unobserved deviation. If zero lies in the support of the deviation given inputs, then the frontier function…
validSharp Concentration Results for Heavy-Tailed Distributions
We obtain concentration and large deviation for the sums of independent and identically distributed random variables with heavy-tailed distributions. Our concentration results are concerned with random variables whose di…
Empirical Risk Minimization with Relative Entropy Regularization: Optimality and Sensitivity Analysis
The optimality and sensitivity of the empirical risk minimization problem with relative entropy regularization (ERM-RER) are investigated for the case in which the reference is a sigma-finite measure instead of a probabi…
SensitivitySmall-time moderate deviations for the randomised Heston model
We extend previous large deviations results for the randomised Heston model to the case of moderate deviations. The proofs involve the G\"artner-Ellis theorem and sharp large deviations tools.