C-STS: Conditional Semantic Textual Similarity
Semantic textual similarity (STS), a cornerstone task in NLP, measures the degree of similarity between a pair of sentences, and has broad application in fields such as information retrieval and natural language understanding. However, sentence similarity can be inherently ambiguous, depending on the specific aspect of interest. We resolve this ambiguity by proposing a novel task called Conditional STS (C-STS) which measures sentences' similarity conditioned on an feature described in natural language (hereon, condition). As an example, the similarity between the sentences "The NBA player shoots a three-pointer." and "A man throws a tennis ball into the air to serve." is higher for the condition "The motion of the ball" (both upward) and lower for "The size of the ball" (one large and one small). C-STS's advantages are two-fold: (1) it reduces the subjectivity and ambiguity of STS and (2) enables fine-grained language model evaluation through diverse natural language conditions. We put several state-of-the-art models to the test, and even those performing well on STS (e.g. SimCSE, Flan-T5, and GPT-4) find C-STS challenging; all with Spearman correlation scores below 50. To encourage a more comprehensive evaluation of semantic similarity and natural language understanding, we make nearly 19K C-STS examples and code available for others to train and test their models.
Code (1)
Tasks
Information RetrievalLanguage Model EvaluationLanguage ModellingNatural Language UnderstandingQuestion AnsweringRetrievalSemantic SimilaritySemantic Textual SimilaritySentenceSentence SimilaritySTSMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Annotating Training Data for Conditional Semantic Textual Similarity Measurement using Large Language Models
Semantic similarity between two sentences depends on the aspects considered between those sentences. To study this phenomenon, Deshpande et al. (2023) proposed the Conditional Semantic Textual Similarity (C-STS) task and…
Semantic Textual SimilaritySemantic SimilarityLinguistically Conditioned Semantic Textual Similarity
Semantic textual similarity (STS) is a fundamental NLP task that measures the semantic similarity between a pair of sentences. In order to reduce the inherent ambiguity posed from the sentences, a recent work called Cond…
Semantic SimilaritySemantic Textual SimilaritySTSNon-parametric Contextual Relationship Learning for Semantic Video Object Segmentation
We propose a novel approach for modeling semantic contextual relationships in videos. This graph-based model enables the learning and propagation of higher-level spatial-temporal contexts to facilitate the semantic label…
Semantic SegmentationVideo Object SegmentationVideo Semantic SegmentationCMLM-CSE: Based on Conditional MLM Contrastive Learning for Sentence Embeddings
Traditional comparative learning sentence embedding directly uses the encoder to extract sentence features, and then passes in the comparative loss function for learning. However, this method pays too much attention to t…
Contrastive LearningLanguage ModelingLanguage ModellingSentence+3CASE -- Condition-Aware Sentence Embeddings for Conditional Semantic Textual Similarity Measurement
The meaning conveyed by a sentence often depends on the context in which it appears. Despite the progress of sentence embedding methods, it remains unclear how to best modify a sentence embedding conditioned on its conte…
Dimensionality ReductionLanguage ModelingLanguage ModellingLarge Language Model+7