Visualizing the PHATE of Neural Networks
Understanding why and how certain neural networks outperform others is key to guiding future development of network architectures and optimization methods. To this end, we introduce a novel visualization algorithm that reveals the internal geometry of such networks: Multislice PHATE (M-PHATE), the first method designed explicitly to visualize how a neural network's hidden representations of data evolve throughout the course of training. We demonstrate that our visualization provides intuitive, detailed summaries of the learning dynamics beyond simple global measures (i.e., validation loss and accuracy), without the need to access validation data. Furthermore, M-PHATE better captures both the dynamics and community structure of the hidden units as compared to visualization based on standard dimensionality reduction methods (e.g., ISOMAP, t-SNE). We demonstrate M-PHATE with two vignettes: continual learning and generalization. In the former, the M-PHATE visualizations display the mechanism of "catastrophic forgetting" which is a major challenge for learning in task-switching contexts. In the latter, our visualizations reveal how increased heterogeneity among hidden units correlates with improved generalization performance. An implementation of M-PHATE, along with scripts to reproduce the figures in this paper, is available at https://github.com/scottgigante/M-PHATE.
Code (1)
Tasks
Continual LearningDimensionality ReductionSimilar Papers 제목 키워드 기반
Multiway Multislice PHATE: Visualizing Hidden Dynamics of RNNs through Training
Recurrent neural networks (RNNs) are a widely used tool for sequential data analysis, however, they are still often seen as black boxes of computation. Understanding the functional principles of these networks is critica…
Exploring the Geometry and Topology of Neural Network Loss Landscapes
Recent work has established clear links between the generalization performance of trained neural networks and the geometry of their loss landscape near the local minima to which they converge. This suggests that qualitat…
Dimensionality ReductionThe use of trade data in the analysis of global phosphate flows
In this paper we present a new method to trace the flows of phosphate from the countries where it is mined to the countries where it is used in agricultural production. We achieve this by combining data on phosphate rock…
Regulation of the Pho Regulon and Reconstruction of the Photosynthetic Apparatus under Phosphate Limitation in Synechocystis sp. PCC 6803
For millennia, cyanobacteria have evolved strategies for acclimatization to dynamic environments- this requires a molecular response system. Phosphate is essential for cellular integrity and metabolism, yet inorganic pho…
Dissecting the Hydrolytic Activities of Sarcoplasmic Reticulum ATPase in the Presence of Acetyl Phosphate
Sarcoplasmic reticulum vesicles and purified Ca$^{2+}$-ATPase hydrolyze acetyl phosphate both in the presence and absence of Ca$^{2+}$. The Ca$^{2+}$-independent activity was fully sensitive to vanadate, insensitive to t…