Information Plane Analysis Visualization in Deep Learning via Transfer Entropy
In a feedforward network, Transfer Entropy (TE) can be used to measure the influence that one layer has on another by quantifying the information transfer between them during training. According to the Information Bottleneck principle, a neural model's internal representation should compress the input data as much as possible while still retaining sufficient information about the output. Information Plane analysis is a visualization technique used to understand the trade-off between compression and information preservation in the context of the Information Bottleneck method by plotting the amount of information in the input data against the compressed representation. The claim that there is a causal link between information-theoretic compression and generalization, measured by mutual information, is plausible, but results from different studies are conflicting. In contrast to mutual information, TE can capture temporal relationships between variables. To explore such links, in our novel approach we use TE to quantify information transfer between neural layers and perform Information Plane analysis. We obtained encouraging experimental results, opening the possibility for further investigations.
Code (0)
등록된 구현이 없습니다.
Tasks
Deep LearningInformation PlaneSimilar Papers 제목 키워드 기반
An Entropy-guided Reinforced Partial Convolutional Network for Zero-Shot Learning
Zero-Shot Learning (ZSL) aims to transfer learned knowledge from observed classes to unseen classes via semantic correlations. A promising strategy is to learn a global-local representation that incorporates global infor…
Generalized Zero-Shot LearningZero-Shot LearningOn the Trajectory of Stochastic Gradient Descent in the Information Plane
Studying the evolution of information theoretic quantities during Stochastic Gradient Descent (SGD) learning of Artificial Neural Networks (ANNs) has gained popularity in recent years. Nevertheless, these type of experi…
Information PlaneInformation Plane Analysis of Deep Neural Networks via Matrix-Based Renyi's Entropy and Tensor Kernels
Analyzing deep neural networks (DNNs) via information plane (IP) theory has gained tremendous attention recently as a tool to gain insight into, among others, their generalization ability. However, it is by no means obvi…
Information PlaneInformation Plane Analysis of Deep Neural Networks via Matrix--Based Renyi's Entropy and Tensor Kernels
Analyzing deep neural networks (DNNs) via information plane (IP) theory has gained tremendous attention recently as a tool to gain insight into, among others, their generalization ability. However, it is by no means obvi…
Information PlaneTransfer entropy applied on EEG in depression reveals aberrated dynamics
We applied transfer entropy analysis on samples of electroencephalogram recorded from patients diagnosed with major depressive disorder and matched healthy controls. This is the first graphical representation of aberrate…
EEGElectroencephalogram (EEG)