Unified Visual-Semantic Embeddings: Bridging Vision and Language With Structured Meaning Representations
We propose the Unified Visual-Semantic Embeddings (Unified VSE) for learning a joint space of visual representation and textual semantics. The model unifies the embeddings of concepts at different levels: objects, attributes, relations, and full scenes. We view the sentential semantics as a combination of different semantic components such as objects and relations; their embeddings are aligned with different image regions. A contrastive learning approach is proposed for the effective learning of this fine-grained alignment from only image-caption pairs. We also present a simple yet effective approach that enforces the coverage of caption embeddings on the semantic components that appear in the sentence. We demonstrate that the Unified VSE outperforms baselines on cross-modal retrieval tasks; the enforcement of the semantic coverage improves the model's robustness in defending text-domain adversarial attacks. Moreover, our model empowers the use of visual cues to accurately resolve word dependencies in novel sentences.
Code (1)
Tasks
Contrastive LearningCross-Modal RetrievalRetrievalSentenceSimilar Papers 제목 키워드 기반
Asymmetric Visual Semantic Embedding Framework for Efficient Vision-Language Alignment
Learning visual semantic similarity is a critical challenge in bridging the gap between images and texts. However, there exist inherent variations between vision and language data, such as information density, i.e., imag…
Semantic SimilaritySemantic Textual SimilarityUnified Representation Space for 3D Visual Grounding
3D visual grounding (3DVG) is a critical task in scene understanding that aims to identify objects in 3D scenes based on text descriptions. However, existing methods rely on separately pre-trained vision and text encoder…
3D visual groundingContrastive LearningScene UnderstandingVisual GroundingBridging Vision and Language Spaces with Assignment Prediction
This paper introduces VLAP, a novel approach that bridges pretrained vision models and large language models (LLMs) to make frozen LLMs understand the visual world. VLAP transforms the embedding space of pretrained visio…
Cross-Modal RetrievalImage CaptioningPredictionQuestion Answering+2MindAlign: Bridging EEG, Vision, and Language for Zero-Shot Visual Decoding
Visual decoding from brain signals is a key challenge at the intersection of computer vision and neuroscience, requiring methods that bridge neural representations and computational models of vision. We introduce a tri-m…
Contrastive LearningTENT: Connect Language Models with IoT Sensors for Zero-Shot Activity Recognition
Recent achievements in language models have showcased their extraordinary capabilities in bridging visual information with semantic language understanding. This leads us to a novel question: can language models connect t…
Activity RecognitionContrastive LearningHuman Activity Recognition