State-Regularized Recurrent Neural Networks
Recurrent neural networks are a widely used class of neural architectures. They have, however, two shortcomings. First, it is difficult to understand what exactly they learn. Second, they tend to work poorly on sequences requiring long-term memorization, despite having this capacity in principle. We aim to address both shortcomings with a class of recurrent networks that use a stochastic state transition mechanism between cell applications. This mechanism, which we term state-regularization, makes RNNs transition between a finite set of learnable states. We evaluate state-regularized RNNs on (1) regular languages for the purpose of automata extraction; (2) nonregular languages such as balanced parentheses, palindromes, and the copy task where external memory is required; and (3) real-word sequence learning tasks for sentiment analysis, visual object recognition, and language modeling. We show that state-regularization (a) simplifies the extraction of finite state automata modeling an RNN's state transition dynamics; (b) forces RNNs to operate more like automata with external memory and less like finite state machines; (c) makes RNNs have better interpretability and explainability.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModelingLanguage ModellingMemorizationObject RecognitionSentiment AnalysisMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Structure Regularized Bidirectional Recurrent Convolutional Neural Network for Relation Classification
Relation classification is an important semantic processing task in the field of natural language processing (NLP). In this paper, we present a novel model, Structure Regularized Bidirectional Recurrent Convolutional Neu…
ClassificationGeneral Classificationnamed-entity-recognitionNamed Entity Recognition+4State-Regularized Recurrent Networks
Recurrent networks are a widely used class of neural architectures. They have, however, two shortcomings. First, it is difficult to understand what exactly they learn. Second, they tend to work poorly on sequences requi…
MemorizationState-Regularized Recurrent Neural Networks to Extract Automata and Explain Predictions
Recurrent neural networks are a widely used class of neural architectures. They have, however, two shortcomings. First, they are often treated as black-box models and as such it is difficult to understand what exactly th…
MemorizationObject RecognitionSentiment AnalysisConvergence of regularized agent-state-based Q-learning in POMDPs
In this paper, we present a framework to understand the convergence of commonly used Q-learning reinforcement learning algorithms in practice. Two salient features of such algorithms are: (i)~the Q-table is recursively u…
Reinforcement LearningStructure Regularized Neural Network for Entity Relation Classification for Chinese Literature Text
Relation classification is an important semantic processing task in the field of natural language processing. In this paper, we propose the task of relation classification for Chinese literature text. A new dataset of Ch…
General ClassificationRelationRelation Classification