Exploring the Effect of Primitives for Compositional Generalization in Vision-and-Language
Compositionality is one of the fundamental properties of human cognition (Fodor & Pylyshyn, 1988). Compositional generalization is critical to simulate the compositional capability of humans, and has received much attention in the vision-and-language (V&L) community. It is essential to understand the effect of the primitives, including words, image regions, and video frames, to improve the compositional generalization capability. In this paper, we explore the effect of primitives for compositional generalization in V&L. Specifically, we present a self-supervised learning based framework that equips V&L methods with two characteristics: semantic equivariance and semantic invariance. With the two characteristics, the methods understand primitives by perceiving the effect of primitive changes on sample semantics and ground-truth. Experimental results on two tasks: temporal video grounding and visual question answering, demonstrate the effectiveness of our framework.
Code (1)
Tasks
Question AnsweringSelf-Supervised LearningVideo GroundingVisual Question AnsweringSimilar Papers 제목 키워드 기반
Exploring Compositionality in Vision Transformers using Wavelet Representations
While insights into the workings of the transformer model have largely emerged by analysing their behaviour on language tasks, this work investigates the representations learnt by the Vision Transformer (ViT) encoder thr…
Representation LearningMulti-Sourced Compositional Generalization in Visual Question Answering
Compositional generalization is the ability of generalizing novel compositions from seen primitives, and has received much attention in vision-and-language (V\&L) recently. Due to the multi-modal nature of V\&L tasks, th…
Question AnsweringVisual Question AnsweringVisual Question Answering (VQA)Exploring Continual Learning of Compositional Generalization in NLI
Compositional Natural Language Inference has been explored to assess the true abilities of neural models to perform NLI. Yet, current evaluations assume models to have full access to all primitive inferences in advance, …
Continual LearningNatural Language InferenceAlgorithmic Primitives and Compositional Geometry of Reasoning in Language Models
How do latent and inference time computations enable large language models (LLMs) to solve multi-step reasoning? We introduce a framework for tracing and steering algorithmic primitives that underlie model reasoning. Our…
Measuring Compositionality in Representation Learning
Many machine learning algorithms represent input data with vector embeddings or discrete codes. When inputs exhibit compositional structure (e.g. objects built from parts or procedures from subroutines), it is natural to…
BIG-bench Machine LearningRepresentation Learning