Tangent Space Sensitivity and Distribution of Linear Regions in ReLU Networks
Recent articles indicate that deep neural networks are efficient models for various learning problems. However they are often highly sensitive to various changes that cannot be detected by an independent observer. As our understanding of deep neural networks with traditional generalization bounds still remains incomplete, there are several measures which capture the behaviour of the model in case of small changes at a specific state. In this paper we consider adversarial stability in the tangent space and suggest tangent sensitivity in order to characterize stability. We focus on a particular kind of stability with respect to changes in parameters that are induced by individual examples without known labels. We derive several easily computable bounds and empirical measures for feed-forward fully connected ReLU (Rectified Linear Unit) networks and connect tangent sensitivity to the distribution of the activation regions in the input space realized by the network. Our experiments suggest that even simple bounds and measures are associated with the empirical generalization gap.
Code (0)
등록된 구현이 없습니다.
Tasks
ArticlesGeneralization BoundsSensitivityMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Estimating a Manifold from a Tangent Bundle Learner
Manifold hypotheses are typically used for tasks such as dimensionality reduction, interpolation, or improving classification performance. In the less common problem of manifold estimation, the task is to characterize th…
Dimensionality ReductionAn Application of the Holonomic Gradient Method to the Neural Tangent Kernel
A holonomic system of linear partial differential equations is, roughly speaking, a system whose solution space is finite dimensional. A distribution that is a solution of a holonomic system is called a holonomic distrib…
Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Models
Task arithmetic has recently emerged as a cost-effective and scalable approach to edit pre-trained models directly in weight space: By adding the fine-tuned weights of different tasks, the model's performance can be impr…
DisentanglementImage ClassificationTask ArithmeticA Tangent Distance Preserving Dimensionality Reduction Algorithm
This paper considers the problem of nonlinear dimensionality reduction. Unlike existing methods, such as LLE, ISOMAP, which attempt to unfold the true manifold in the low dimensional space, our algorithm tries to preserv…
Dimensionality ReductionCausal Portfolio Optimization: Principles and Sensitivity-Based Solutions
Fundamental and necessary principles for achieving efficient portfolio optimization based on asset and diversification dynamics are presented. The Commonality Principle is a necessary and sufficient condition for identif…
Portfolio OptimizationSensitivity