Tree-structured composition in neural networks without tree-structured architectures
Tree-structured neural networks encode a particular tree geometry for a sentence in the network design. However, these models have at best only slightly outperformed simpler sequence-based models. We hypothesize that neural sequence models like LSTMs are in fact able to discover and implicitly use recursive compositional structure, at least for tasks with clear cues to that structure in the data. We demonstrate this possibility using an artificial data task for which recursive compositional structure is crucial, and find an LSTM-based sequence model can indeed learn to exploit the underlying tree structure. However, its performance consistently lags behind that of tree models, even on large training sets, suggesting that tree-structured models are more effective at exploiting recursive structure.
Code (1)
Tasks
SentenceSimilar Papers 제목 키워드 기반
Tensor Decompositions in Recursive Neural Networks for Tree-Structured Data
The paper introduces two new aggregation functions to encode structural knowledge from tree-structured data. They leverage the Canonical and Tensor-Train decompositions to yield expressive context aggregation while limit…
General ClassificationMultiplicative Tree-Structured Long Short-Term Memory Networks for Semantic Representations
Tree-structured LSTMs have shown advantages in learning semantic representations by exploiting syntactic information. Most existing methods model tree structures by bottom-up combinations of constituent nodes using the s…
Abstract Meaning RepresentationLearning Semantic RepresentationsMachine TranslationRelation Extraction+3Characterizing Intrinsic Compositionality in Transformers with Tree Projections
When trained on language data, do transformers learn some arbitrary computation that utilizes the full capacity of the architecture or do they learn a simpler, tree-like computation, hypothesized to underlie compositiona…
SentenceClustering Tree-structured Data on Manifold
Tree-structured data usually contain both topological and geometrical information, and are necessarily considered on manifold instead of Euclidean space for appropriate data parameterization and analysis. In this study, …
AttributeClusteringTree DecompositionA Framework for End-to-End Learning on Semantic Tree-Structured Data
While learning models are typically studied for inputs in the form of a fixed dimensional feature vector, real world data is rarely found in this form. In order to meet the basic requirement of traditional learning model…
Reinforcement Learning