Evaluating Multimodal Representations on Sentence Similarity: vSTS, Visual Semantic Textual Similarity Dataset
In this paper we introduce vSTS, a new dataset for measuring textual similarity of sentences using multimodal information. The dataset is comprised by images along with its respectively textual captions. We describe the dataset both quantitatively and qualitatively, and claim that it is a valid gold standard for measuring automatic multimodal textual similarity systems. We also describe the initial experiments combining the multimodal information.
Code (0)
등록된 구현이 없습니다.
Tasks
Semantic Textual SimilaritySentenceSentence SimilarityvalidSimilar Papers 제목 키워드 기반
Evaluating Multimodal Representations on Visual Semantic Textual Similarity
The combination of visual and textual representations has produced excellent results in tasks such as image captioning and visual question answering, but the inference capabilities of multimodal representations are large…
BenchmarkingImage CaptioningNatural Language InferenceQuestion Answering+3SentEval: An Evaluation Toolkit for Universal Sentence Representations
We introduce SentEval, a toolkit for evaluating the quality of universal sentence representations. SentEval encompasses a variety of tasks, including binary and multi-class classification, natural language inference and …
General ClassificationMulti-class ClassificationNatural Language InferenceSentence+1Instance-aware Image and Sentence Matching with Selective Multimodal LSTM
Effective image and sentence matching depends on how to well measure their global visual-semantic similarity. Based on the observation that such a global similarity arises from a complex aggregation of multiple local sim…
Semantic SimilaritySemantic Textual SimilaritySentenceLearning semantic sentence representations from visually grounded language without lexical knowledge
Current approaches to learning semantic representations of sentences often use prior word-level knowledge. The current study aims to leverage visual information in order to capture sentence level semantics without the ne…
Grounded language learningLearning Semantic RepresentationsRetrievalSemantic Similarity+5Contrastive Visual Semantic Pretraining Magnifies the Semantics of Natural Language Representations
We examine the effects of contrastive visual semantic pretraining by comparing the geometry and semantic properties of contextualized English language representations formed by GPT-2 and CLIP, a zero-shot multimodal imag…
Image CaptioningSemantic Textual SimilaritySentenceSentence Embeddings+1