Disentangled Sequence to Sequence Learning for Compositional Generalization
There is mounting evidence that existing neural network models, in particular the very popular sequence-to-sequence architecture, struggle to systematically generalize to unseen compositions of seen components. We demonstrate that one of the reasons hindering compositional generalization relates to representations being entangled. We propose an extension to sequence-to-sequence models which encourage disentanglement by adaptively re-encoding (at each time step) the source input. Specifically, we condition the source representations on the newly decoded target context which makes it easier for the encoder to exploit specialized information for each prediction rather than capturing it all in a single forward pass. Experimental results on semantic parsing and machine translation empirically show that our proposal delivers more disentangled representations and better generalization.
Code (0)
등록된 구현이 없습니다.
Tasks
DisentanglementMachine TranslationSemantic ParsingTranslationSimilar Papers 제목 키워드 기반
Real-World Compositional Generalization with Disentangled Sequence-to-Sequence Learning
Compositional generalization is a basic mechanism in human language learning, which current neural networks struggle with. A recently proposed Disentangled sequence-to-sequence model (Dangle) shows promising generalizati…
Machine TranslationDisentangled Sequence to Sequence Learning for Compositional Generalization
There is mounting evidence that existing neural network models, in particular the very popular sequence-to-sequence architecture, struggle to systematically generalize to unseen compositions of seen components. We demons…
DisentanglementMachine TranslationSemantic ParsingTranslationCompositional Generalization via Semantic Tagging
Although neural sequence-to-sequence models have been successfully applied to semantic parsing, they fail at compositional generalization, i.e., they are unable to systematically generalize to unseen compositions of seen…
Semantic ParsingTAGCompositional Generalization Requires Compositional Parsers
A rapidly growing body of research on compositional generalization investigates the ability of a semantic parser to dynamically recombine linguistic elements seen in training into unseen sequences. We present a systemati…
On Provable Length and Compositional Generalization
Out-of-distribution generalization capabilities of sequence-to-sequence models can be studied from the lens of two crucial forms of generalization: length generalization -- the ability to generalize to longer sequences t…
DiversityOut-of-Distribution GeneralizationState Space Models