Do LSTMs Learn Compositionally?
LSTM-based language models exhibit compositionality in their representations, but how this behavior emerges over the course of training has not been explored. Analyzing synthetic data experiments with contextual decomposition, we find that LSTMs learn long-range dependencies compositionally by building them from shorter constituents during training.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Recursive Neural Networks with Bottlenecks Diagnose (Non-)Compositionality
A recent line of work in NLP focuses on the (dis)ability of models to generalise compositionally for artificial languages. However, when considering natural language tasks, the data involved is not strictly, or locally, …
Sentiment AnalysisSentiment ClassificationWhen can transformers compositionally generalize in-context?
Many tasks can be composed from a few independent components. This gives rise to a combinatorial explosion of possible tasks, only some of which might be encountered during training. Under what circumstances can transfor…
Propositional Logic for Probing Generalization in Neural Networks
The extent to which neural networks are able to acquire and represent symbolic rules remains a key topic of research and debate. Much current work focuses on the impressive capabilities of large language models, as well …
NegationDetecting Compositionally Out-of-Distribution Examples in Semantic Parsing
While neural networks are ubiquitous in state-of-the-art semantic parsers, it has been shown that most standard models suffer from dramatic performance losses when faced with compositionally out-of-distribution (OOD) dat…
Out of Distribution (OOD) DetectionSemantic ParsingCompositional generalization through meta sequence-to-sequence learning
People can learn a new concept and use it compositionally, understanding how to "blicket twice" after learning how to "blicket." In contrast, powerful sequence-to-sequence (seq2seq) neural networks fail such tests of com…