Learning compositionally through attentive guidance
While neural network models have been successfully applied to domains that require substantial generalisation skills, recent studies have implied that they struggle when solving the task they are trained on requires inferring its underlying compositional structure. In this paper, we introduce Attentive Guidance, a mechanism to direct a sequence to sequence model equipped with attention to find more compositional solutions. We test it on two tasks, devised precisely to assess the compositional capabilities of neural models, and we show that vanilla sequence to sequence models with attention overfit the training distribution, while the guided versions come up with compositional solutions that fit the training and testing distributions almost equally well. Moreover, the learned solutions generalise even in cases where the training and testing distributions strongly diverge. In this way, we demonstrate that sequence to sequence models are capable of finding compositional solutions without requiring extra components. These results helps to disentangle the causes for the lack of systematic compositionality in neural networks, which can in turn fuel future work.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Compositionally Equivariant Representation Learning
Deep learning models often need sufficient supervision (i.e. labelled data) in order to be trained effectively. By contrast, humans can swiftly learn to identify important anatomy in medical images like MRI and CT scans,…
AnatomyImage SegmentationMedical Image SegmentationRepresentation Learning+1Attentive Knowledge-aware Graph Convolutional Networks with Collaborative Guidance for Personalized Recommendation
To alleviate data sparsity and cold-start problems of traditional recommender systems (RSs), incorporating knowledge graphs (KGs) to supplement auxiliary information has attracted considerable attention recently. However…
Click-Through Rate PredictionKnowledge-Aware RecommendationKnowledge GraphsRecommendation SystemsAttentive Eraser: Unleashing Diffusion Model's Object Removal Potential via Self-Attention Redirection Guidance
Recently, diffusion models have emerged as promising newcomers in the field of generative models, shining brightly in image generation. However, when employed for object removal tasks, they still encounter issues such as…
Image GenerationObjectCompositional generalization through meta sequence-to-sequence learning
People can learn a new concept and use it compositionally, understanding how to "blicket twice" after learning how to "blicket." In contrast, powerful sequence-to-sequence (seq2seq) neural networks fail such tests of com…
Effect of Adaptive and Fixed Shared Steering Control on Distracted Driver Behavior
Driver distraction is a well-known cause for traffic collisions worldwide. Studies have indicated that shared steering control, which actively provides haptic guidance torque on the steering wheel, effectively improves t…
Steering Control