Pointwise Paraphrase Appraisal is Potentially Problematic
The prevailing approach for training and evaluating paraphrase identification models is constructed as a binary classification problem: the model is given a pair of sentences, and is judged by how accurately it classifies pairs as either paraphrases or non-paraphrases. This pointwise-based evaluation method does not match well the objective of most real world applications, so the goal of our work is to understand how models which perform well under pointwise evaluation may fail in practice and find better methods for evaluating paraphrase identification models. As a first step towards that goal, we show that although the standard way of fine-tuning BERT for paraphrase identification by pairing two sentences as one sequence results in a model with state-of-the-art performance, that model may perform poorly on simple tasks like identifying pairs with two identical sentences. Moreover, we show that these models may even predict a pair of randomly-selected sentences with higher paraphrase score than a pair of identical ones.
Code (0)
등록된 구현이 없습니다.
Tasks
Binary ClassificationParaphrase IdentificationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
MIPA: Mutual Information Based Paraphrase Acquisition via Bilingual Pivoting
We present a pointwise mutual information (PMI)-based approach to formalize paraphrasability and propose a variant of PMI, called MIPA, for the paraphrase acquisition. Our paraphrase acquisition method first acquires lex…
Learning Word EmbeddingsSemantic Textual SimilarityWord AlignmentWord EmbeddingsAffective Natural Language Generation of Event Descriptions through Fine-grained Appraisal Conditions
Models for affective text generation have shown a remarkable progress, but they commonly rely only on basic emotion theories or valance/arousal values as conditions. This is appropriate when the goal is to create explici…
Text GenerationResting-State fingerprints of Acceptance and Reappraisal. The role of Sensorimotor, Executive and Affective networks
Acceptance and Reappraisal are considered adaptive emotion regulation strategies. While previous studies have explored the neural underpinnings of these strategies using task based fMRI and sMRI, a gap exists in the lite…
Functional ConnectivityA critical reappraisal of predicting suicidal ideation using fMRI
For many psychiatric disorders, neuroimaging offers a potential for revolutionizing diagnosis, and potentially treatment, by providing access to preverbal mental processes. In their study "Machine learning of neural repr…
BIG-bench Machine LearningDensity-Ratio Based Personalised Ranking from Implicit Feedback
Learning from implicit user feedback is challenging as we can only observe positive samples but never access negative ones. Most conventional methods cope with this issue by adopting a pairwise ranking approach with nega…
Density Ratio Estimation