Modeling Compositionality with Multiplicative Recurrent Neural Networks
We present the multiplicative recurrent neural network as a general model for compositional meaning in language, and evaluate it on the task of fine-grained sentiment analysis. We establish a connection to the previously investigated matrix-space models for compositionality, and show they are special cases of the multiplicative recurrent net. Our experiments show that these models perform comparably or better than Elman-type additive recurrent neural networks and outperform matrix-space models on a standard fine-grained sentiment analysis corpus. Furthermore, they yield comparable results to structural deep models on the recently published Stanford Sentiment Treebank without the need for generating parse trees.
Code (0)
등록된 구현이 없습니다.
Tasks
Sentiment AnalysisSimilar Papers 제목 키워드 기반
Multiplicative Models for Recurrent Language Modeling
Recently, there has been interest in multiplicative recurrent neural networks for language modeling. Indeed, simple Recurrent Neural Networks (RNNs) encounter difficulties recovering from past mistakes when generating se…
Language ModelingLanguage ModellingDeep Recursive Neural Networks for Compositionality in Language
Recursive neural networks comprise a class of architecture that can operate on structured input. They have been previously successfully applied to model compositionality in natural language using parse-tree-based structu…
Sentiment AnalysisSentiment ClassificationLanguage Modeling with a General Second-Order RNN
Different Recurrent Neural Network (RNN) architectures update their state in different manners as the input sequence is processed. RNNs including a multiplicative interaction between their current state and the current i…
Language ModelingLanguage ModellingLearning compositional functions via multiplicative weight updates
Compositionality is a basic structural feature of both biological and artificial neural networks. Learning compositional functions via gradient descent incurs well known problems like vanishing and exploding gradients, m…
LEMMAGated recurrent neural networks discover attention
Recent architectural developments have enabled recurrent neural networks (RNNs) to reach and even surpass the performance of Transformers on certain sequence modeling tasks. These modern RNNs feature a prominent design p…
In-Context Learning