Neural network interpretation using descrambler groups
The lack of interpretability and trust is a much-criticised feature of deep neural networks. In fully connected nets, the signalling between inner layers is scrambled because backpropagation training does not require perceptrons to be arranged in any particular order. The result is a black box; this problem is particularly severe in scientific computing and digital signal processing (DSP), where neutral nets perform abstract mathematical transformations that do not reduce to features or concepts. We present here a group-theoretical procedure that attempts to bring inner layer signalling into a human-readable form, the assumption being that this form exists and has identifiable and quantifiable features - for example, smoothness or locality. We applied the proposed method to DEERNet (a DSP network used in electron spin resonance) and managed to descramble it. We found considerable internal sophistication: the network spontaneously invents a bandpass filter, a notch filter, a frequency axis rectifier, frequency division multiplexing, group embedding, spectral filtering regularisation, and a Fourier-like transform that maps harmonic functions into Chebyshev polynomials - in ten minutes of unattended training from a random initial guess.
Code (0)
등록된 구현이 없습니다.
Tasks
Network InterpretationSimilar Papers 제목 키워드 기반
On the limits of neural network explainability via descrambling
We characterize the exact solutions to neural network descrambling--a mathematical model for explaining the fully connected layers of trained neural networks (NNs). By reformulating the problem to the minimization of the…
Graph MatchingOperator learningModel-agnostic Feature Importance and Effects with Dependent Features -- A Conditional Subgroup Approach
The interpretation of feature importance in machine learning models is challenging when features are dependent. Permutation feature importance (PFI) ignores such dependencies, which can cause misleading interpretations d…
Feature ImportanceA World of Difference: Divergent Word Interpretations among People
Divergent word usages reflect differences among people. In this paper, we present a novel angle for studying word usage divergence -- word interpretations. We propose an approach that quantifies semantic differences in i…
Self-learning photonic signal processor with an optical neural network chip
Photonic signal processing is essential in the optical communication and optical computing. Numerous photonic signal processors have been proposed, but most of them exhibit limited reconfigurability and automaticity. A f…
Self-LearningMachines Learn Number Fields, But How? The Case of Galois Groups
By applying interpretable machine learning methods such as decision trees, we study how simple models can classify the Galois groups of Galois extensions over $\mathbb{Q}$ of degrees 4, 6, 8, 9, and 10, using Dedekind ze…
Interpretable Machine Learning