Computing Linear Regions in Neural Networks with Skip Connections
Neural networks are important tools in machine learning. Representing piecewise linear activation functions with tropical arithmetic enables the application of tropical geometry. Algorithms are presented to compute regions where the neural networks are linear maps. Through computational experiments, we provide insights on the difficulty to train neural networks, in particular on the problems of overfitting and on the benefits of skip connections.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
A General Computational Framework to Measure the Expressiveness of Complex Networks using a Tight Upper Bound of Linear Regions
The expressiveness of deep neural network (DNN) is a perspective to understand the surprising performance of DNN. The number of linear regions, i.e. pieces that a piece-wise-linear function represented by a DNN, is gener…
Efficient Skip Connections Realization for Secure Inference on Encrypted Data
Homomorphic Encryption (HE) is a cryptographic tool that allows performing computation under encryption, which is used by many privacy-preserving machine learning solutions, for example, to perform secure classification.…
Privacy PreservingA General Computational Framework to Measure the Expressiveness of Complex Networks Using a Tighter Upper Bound of Linear Regions
The expressiveness of deep neural network (DNN) is a perspective to understandthe surprising performance of DNN. The number of linear regions, i.e. pieces thata piece-wise-linear function represented by a DNN, is general…
Select, Attend, and Transfer: Light, Learnable Skip Connections
Skip connections in deep networks have improved both segmentation and classification performance by facilitating the training of deeper network architectures, and reducing the risks for vanishing gradients. They equip en…
DecoderSegmentationOptimization of Graph Neural Networks: Implicit Acceleration by Skip Connections and More Depth
Graph Neural Networks (GNNs) have been studied through the lens of expressive power and generalization. However, their optimization properties are less well understood. We take the first step towards analyzing GNN traini…