Real-World Compositional Generalization with Disentangled Sequence-to-Sequence Learning
Compositional generalization is a basic mechanism in human language learning, which current neural networks struggle with. A recently proposed Disentangled sequence-to-sequence model (Dangle) shows promising generalization capability by learning specialized encodings for each decoding step. We introduce two key modifications to this model which encourage more disentangled representations and improve its compute and memory efficiency, allowing us to tackle compositional generalization in a more realistic setting. Specifically, instead of adaptively re-encoding source keys and values at each time step, we disentangle their representations and only re-encode keys periodically, at some interval. Our new architecture leads to better generalization performance across existing tasks and datasets, and a new machine translation benchmark which we create by detecting naturally occurring compositional patterns in relation to a training set. We show this methodology better emulates real-world requirements than artificial challenges.
Code (0)
등록된 구현이 없습니다.
Tasks
Machine TranslationSimilar Papers 제목 키워드 기반
Disentangled Sequence to Sequence Learning for Compositional Generalization
There is mounting evidence that existing neural network models, in particular the very popular sequence-to-sequence architecture, struggle to systematically generalize to unseen compositions of seen components. We demons…
DisentanglementMachine TranslationSemantic ParsingTranslationDisentangled Sequence to Sequence Learning for Compositional Generalization
There is mounting evidence that existing neural network models, in particular the very popular sequence-to-sequence architecture, struggle to systematically generalize to unseen compositions of seen components. We demon…
DisentanglementMachine TranslationSemantic ParsingTranslationZero-Shot Policy Transfer with Disentangled Task Representation of Meta-Reinforcement Learning
Humans are capable of abstracting various tasks as different combinations of multiple attributes. This perspective of compositionality is vital for human rapid learning and adaption since previous experiences from relate…
DisentanglementMeta Reinforcement Learningreinforcement-learningReinforcement Learning (RL)Compositional Generalization via Semantic Tagging
Although neural sequence-to-sequence models have been successfully applied to semantic parsing, they fail at compositional generalization, i.e., they are unable to systematically generalize to unseen compositions of seen…
Semantic ParsingTAGA causal view of compositional zero-shot recognition
People easily recognize new visual categories that are new combinations of known components. This compositional generalization capacity is critical for learning in real-world domains like vision and language because the …
AttributeCompositional Zero-Shot LearningZero-Shot Learning