NeuroX: A Toolkit for Analyzing Individual Neurons in Neural Networks
We present a toolkit to facilitate the interpretation and understanding of neural network models. The toolkit provides several methods to identify salient neurons with respect to the model itself or an external task. A user can visualize selected neurons, ablate them to measure their effect on the model accuracy, and manipulate them to control the behavior of the model at the test time. Such an analysis has a potential to serve as a springboard in various research directions, such as understanding the model, better architectural choices, model distillation and controlling data biases.
Code (2)
Similar Papers 제목 키워드 기반
NeuroX Library for Neuron Analysis of Deep NLP Models
Neuron analysis provides insights into how knowledge is structured in representations and discovers the role of neurons in the network. In addition to developing an understanding of our models, neuron analysis enables va…
Domain AdaptationWhat Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Models
Despite the remarkable evolution of deep neural networks in natural language processing (NLP), their interpretability remains a challenge. Previous work largely focused on what these models learn at the representation le…
Language ModelingLanguage ModellingMachine TranslationNMT+1Fine-grained Interpretation and Causation Analysis in Deep NLP Models
This paper is a write-up for the tutorial on "Fine-grained Interpretation and Causation Analysis in Deep NLP Models" that we are presenting at NAACL 2021. We present and discuss the research work on interpreting fine-gra…
Domain AdaptationDiscovering Salient Neurons in Deep NLP Models
While a lot of work has been done in understanding representations learned within deep NLP models and what knowledge they capture, little attention has been paid towards individual neurons. We present a technique called …
Transfer LearningLM Transparency Tool: Interactive Tool for Analyzing Transformer Language Models
We present the LM Transparency Tool (LM-TT), an open-source interactive toolkit for analyzing the internal workings of Transformer-based language models. Differently from previously existing tools that focus on isolated …
Decision Making