Compositional Mixture Representations for Vision and Text
Learning a common representation space between vision and language allows deep networks to relate objects in the image to the corresponding semantic meaning. We present a model that learns a shared Gaussian mixture representation imposing the compositionality of the text onto the visual domain without having explicit location supervision. By combining the spatial transformer with a representation learning approach we learn to split images into separately encoded patches to associate visual and textual representations in an interpretable manner. On variations of MNIST and CIFAR10, our model is able to perform weakly supervised object detection and demonstrates its ability to extrapolate to unseen combination of objects.
Code (0)
등록된 구현이 없습니다.
Tasks
object-detectionObject DetectionRepresentation LearningWeakly Supervised Object DetectionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Illiterate DALL$\cdot$E Learns to Compose
DALL$\cdot$E has shown an impressive ability of composition-based systematic generalization in image generation. This is possible because it utilizes the dataset of text-image pairs where the text provides the source of …
DecoderImage GenerationSystematic GeneralizationWhen and How Does CLIP Enable Domain and Compositional Generalization?
The remarkable generalization performance of contrastive vision-language models like CLIP is often attributed to the diversity of their training distributions. However, key questions remain unanswered: Can CLIP generaliz…
DiversityDomain GeneralizationNot Only Text: Exploring Compositionality of Visual Representations in Vision-Language Models
Vision-Language Models (VLMs) learn a shared feature space for text and images, enabling the comparison of inputs of different modalities. While prior works demonstrated that VLMs organize natural language representation…
Aitchison Embeddings for Learning Compositional Graph Representations
Representation learning is central to graph machine learning, powering tasks such as link prediction and node classification. However, most graph embeddings are hard to interpret, offering limited insight into how learne…
Representation LearningNode ClassificationLink PredictionGraph EmbeddingPreserving Multi-Modal Capabilities of Pre-trained VLMs for Improving Vision-Linguistic Compositionality
In this paper, we propose a new method to enhance compositional understanding in pre-trained vision and language models (VLMs) without sacrificing performance in zero-shot multi-modal tasks. Traditional fine-tuning appro…