TropEx: An Algorithm for Extracting Linear Terms in Deep Neural Networks
Deep neural networks with rectified linear (ReLU) activations are piecewise linear functions, where hyperplanes partition the input space into an astronomically high number of linear regions. Previous work focused on counting linear regions to measure the network's expressive power and on analyzing geometric properties of the hyperplane configurations. In contrast, we aim to understand the impact of the linear terms on network performance, by examining the information encoded in their coefficients. To this end, we derive TropEx, a nontrivial tropical algebra-inspired algorithm to systematically extract linear terms based on data. Applied to convolutional and fully-connected networks, our algorithm uncovers significant differences in how the different networks utilize linear regions for generalization. This underlines the importance of systematic linear term exploration, to better understand generalization in neural networks trained with complex data sets.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Randomised Algorithm for Feature Selection and Classification
We here introduce a novel classification approach adopted from the nonlinear model identification framework, which jointly addresses the feature selection and classifier design tasks. The classifier is constructed as a p…
Classificationfeature selectionGeneral ClassificationExtracting Latent State Representations with Linear Dynamics from Rich Observations
Recently, many reinforcement learning techniques were shown to have provable guarantees in the simple case of linear dynamics, especially in problems like linear quadratic regulators. However, in practice, many reinforce…
Positionreinforcement-learningReinforcement LearningReinforcement Learning (RL)Provably Extracting the Features from a General Superposition
It is widely believed that complex machine learning models generally encode features through linear representations. This is the foundational hypothesis behind a vast body of work on interpretability. A key challenge tow…
Equation discovery framework EPDE: Towards a better equation discovery
Equation discovery methods hold promise for extracting knowledge from physics-related data. However, existing approaches often require substantial prior information that significantly reduces the amount of knowledge extr…
Equation DiscoverySemi-Supervised Technical Term Tagging With Minimal User Feedback
In this paper, we address the problem of extracting technical terms automatically from an unannotated corpus. We introduce a technology term tagger that is based on Liblinear Support Vector Machines and employs linguisti…
Dependency ParsingLanguage ModellingWord Sense Disambiguation