The Many Senses of Visual Similarity: A Text-Prompted Image Perceptual Metric
Human visual similarity judgments are context-dependent. For example, two images may be similar in shape but distinct in color. Existing perceptual similarity metrics, however, collapse these nuances into a single scalar value, offering no mechanism to condition on specific aspects. To bridge this gap, we introduce a large-scale dataset of human similarity judgments over image triplets, where each triplet is annotated across multiple, free-form semantic aspects of similarity. Benchmarking a broad range of frontier vision-language models (VLMs) reveals a considerable performance gap compared to human annotators' consensus. Leveraging our data, we fine-tune a VLM to produce our Text-Prompted Image Perceptual Similarity (TPIPS) metric, capturing multiple senses of visual similarity depending on the specified text prompt. We demonstrate that TPIPS aligns more closely with human perception and generalizes reliably beyond the training distribution. Finally, we show that TPIPS unlocks new capabilities in text-guided retrieval, compositional search, and the fine-grained evaluation of generative models. Our code, data, and trained models are at https://peterwang512.github.io/TPIPS
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Sense Discovery via Co-Clustering on Images and Text
We present a co-clustering framework that can be used to discover multiple semantic and visual senses of a given Noun Phrase (NP). Unlike traditional clustering approaches which assume a one-to-one mapping between the cl…
ClusteringA Quadratic 0-1 Programming Approach for Word Sense Disambiguation
Word Sense Disambiguation (WSD) is the task to determine the sense of an ambiguous word in a given context. Previous approaches for WSD have focused on supervised and knowledge-based methods, but inter-sense interactions…
Combinatorial OptimizationWord Sense DisambiguationWord SimilarityVisualising WordNet Embeddings: some preliminary results
AutoExtend is a method for learning unambiguous vector embeddings for word senses. We visualise these word embeddings with t-SNE, which further compresses the vectors to the x,y plane. We show that the t-SNE co-ordinates…
Semantic SimilaritySemantic Textual SimilarityWord EmbeddingsPolyUCOMP-CORE\_TYPED: Computing Semantic Textual Similarity using Overlapped Senses
BERT Has Uncommon Sense: Similarity Ranking for Word Sense BERTology
An important question concerning contextualized word embedding (CWE) models like BERT is how well they can represent different word senses, especially those in the long tail of uncommon senses. Rather than build a WSD sy…
Retrieval