RNNbow: Visualizing Learning via Backpropagation Gradients in Recurrent Neural Networks
We present RNNbow, an interactive tool for visualizing the gradient flow during backpropagation training in recurrent neural networks. RNNbow is a web application that displays the relative gradient contributions from Recurrent Neural Network (RNN) cells in a neighborhood of an element of a sequence. We describe the calculation of backpropagation through time (BPTT) that keeps track of itemized gradients, or gradient contributions from one element of a sequence to previous elements of a sequence. By visualizing the gradient, as opposed to activations, RNNbow offers insight into how the network is learning. We use it to explore the learning of an RNN that is trained to generate code in the C programming language. We show how it uncovers insights into the vanishing gradient as well as the evolution of training as the RNN works its way through a corpus.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
A Gentle Tutorial of Recurrent Neural Network with Error Backpropagation
We describe recurrent neural networks (RNNs), which have attracted great attention on sequential tasks, such as handwriting recognition, speech recognition and image to text. However, compared to general feedforward neur…
Handwriting RecognitionImage to textspeech-recognitionSpeech RecognitionBackpropagation Through Time For Networks With Long-Term Dependencies
Backpropagation through time (BPTT) is a technique of updating tuned parameters within recurrent neural networks (RNNs). Several attempts at creating such an algorithm have been made including: Nth Ordered Approximations…
Gradient-free training of recurrent neural networks
Recurrent neural networks are a successful neural architecture for many time-dependent problems, including time series analysis, forecasting, and modeling of dynamical systems. Training such networks with backpropagation…
Time SeriesTime Series AnalysisTime Series ForecastingOn Training Recurrent Networks with Truncated Backpropagation Through Time in Speech Recognition
Recurrent neural networks have been the dominant models for many speech and language processing tasks. However, we understand little about the behavior and the class of functions recurrent networks can realize. Moreover,…
speech-recognitionSpeech RecognitionAlternating Synthetic and Real Gradients for Neural Language Modeling
Training recurrent neural networks (RNNs) with backpropagation through time (BPTT) has known drawbacks such as being difficult to capture longterm dependencies in sequences. Successful alternatives to BPTT have not yet b…
Language ModelingLanguage Modelling