Exposing the Functionalities of Neurons for Gated Recurrent Unit Based Sequence-to-Sequence Model
The goal of this paper is to report certain scientific discoveries about a Seq2Seq model. It is known that analyzing the behavior of RNN-based models at the neuron level is considered a more challenging task than analyzing a DNN or CNN models due to their recursive mechanism in nature. This paper aims to provide neuron-level analysis to explain why a vanilla GRU-based Seq2Seq model without attention can achieve token-positioning. We found four different types of neurons: storing, counting, triggering, and outputting and further uncover the mechanism for these neurons to work together in order to produce the right token in the right position.
Code (0)
등록된 구현이 없습니다.
Tasks
PositionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
A bio-inspired bistable recurrent cell allows for long-lasting memory
Recurrent neural networks (RNNs) provide state-of-the-art performances in a wide variety of tasks that require memory. These performances can often be achieved thanks to gated recurrent cells such as gated recurrent unit…
Time Series AnalysisBayesian Sparsification of Gated Recurrent Neural Networks
Bayesian methods have been successfully applied to sparsify weights of neural networks and to remove structure units from the networks, e. g. neurons. We apply and further develop this approach for gated recurrent archit…
Structured Sparsification of Gated Recurrent Neural Networks
Recently, a lot of techniques were developed to sparsify the weights of neural networks and to remove networks' structure units, e.g. neurons. We adjust the existing sparsification approaches to the gated recurrent archi…
Language ModelingLanguage Modellingtext-classificationText ClassificationConvolutional Neural Networks with Gated Recurrent Connections
The convolutional neural network (CNN) has become a basic model for solving many computer vision problems. In recent years, a new class of CNNs, recurrent convolution neural network (RCNN), inspired by abundant recurrent…
object-detectionObject DetectionObject RecognitionScene Text RecognitionGRSN: Gated Recurrent Spiking Neurons for POMDPs and MARL
Spiking neural networks (SNNs) are widely applied in various fields due to their energy-efficient and fast-inference capabilities. Applying SNNs to reinforcement learning (RL) can significantly reduce the computational r…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)