Dive into Layers: Neural Network Capacity Bounding using Algebraic Geometry
The empirical results suggest that the learnability of a neural network is directly related to its size. To mathematically prove this, we borrow a tool in topological algebra: Betti numbers to measure the topological geometric complexity of input data and the neural network. By characterizing the expressive capacity of a neural network with its topological complexity, we conduct a thorough analysis and show that the network's expressive capacity is limited by the scale of its layers. Further, we derive the upper bounds of the Betti numbers on each layer within the network. As a result, the problem of architecture selection of a neural network is transformed to determining the scale of the network that can represent the input data complexity. With the presented results, the architecture selection of a fully connected network boils down to choosing a suitable size of the network such that it equips the Betti numbers that are not smaller than the Betti numbers of the input data. We perform the experiments on a real-world dataset MNIST and the results verify our analysis and conclusion. The code is publicly available.
Code (1)
Similar Papers 제목 키워드 기반
Paying Attention to Facts: Quantifying the Knowledge Capacity of Attention Layers
In this paper, we investigate the ability of single-layer attention-only transformers (i.e. attention layers) to memorize facts contained in databases from a linear-algebraic perspective. We associate with each database …
Physics-Guided Dimension Reduction for Simulation-Free Operator Learning of Stiff Differential-Algebraic Systems
Neural surrogates for stiff differential-algebraic equations (DAEs) face two barriers: soft-constraint methods leave algebraic residuals that stiffness amplifies into errors, and hard-constraint methods require trajector…
Out-of-Distribution DetectionA Unified Algebraic Perspective on Lipschitz Neural Networks
Important research efforts have focused on the design and training of neural networks with a controlled Lipschitz constant. The goal is to increase and sometimes guarantee the robustness against adversarial attacks. Rece…
image-classificationImage ClassificationProvable Adversarial DefenseHosting Capacity Approach
This chapter proposes an evolved concept of "hosting capacity" using the term of "feasible region". Through converting the grid model into a more compact one, "hosting capacity region" not only is promising to further ex…
Optimal Robust Network Design: Formulations and Algorithms for Maximizing Algebraic Connectivity
This paper focuses on designing edge-weighted networks, whose robustness is characterized by maximizing algebraic connectivity, or the second smallest eigenvalue of the Laplacian matrix. This problem is motivated by coop…
Autonomous VehiclesPosition