Social Bias in Elicited Natural Language Inferences
We analyze the Stanford Natural Language Inference (SNLI) corpus in an investigation of bias and stereotyping in NLP data. The SNLI human-elicitation protocol makes it prone to amplifying bias and stereotypical associations, which we demonstrate statistically (using pointwise mutual information) and with qualitative examples.
Code (1)
Tasks
Language ModelingLanguage ModellingNatural Language InferenceWord EmbeddingsSimilar Papers 제목 키워드 기반
Hypothesis-only Biases in Large Language Model-Elicited Natural Language Inference
We test whether replacing crowdsource workers with LLMs to write Natural Language Inference (NLI) hypotheses similarly results in annotation artifacts. We recreate a portion of the Stanford NLI corpus using GPT-4, Llama-…
Language ModelingLanguage ModellingLarge Language ModelNatural Language InferenceEvaluating Gender Bias of Pre-trained Language Models in Natural Language Inference by Considering All Labels
Discriminatory gender biases have been found in Pre-trained Language Models (PLMs) for multiple languages. In Natural Language Inference (NLI), existing bias evaluation methods have focused on the prediction results of o…
AllNatural Language InferenceOn Measuring and Mitigating Biased Inferences of Word Embeddings
Word embeddings carry stereotypical connotations from the text they are trained on, which can lead to invalid inferences in downstream models that rely on them. We use this observation to design a mechanism for measuring…
Natural Language InferenceWord EmbeddingsVIGNETTE: Socially Grounded Bias Evaluation for Vision-Language Models
While bias in large language models (LLMs) is well-studied, similar concerns in vision-language models (VLMs) have received comparatively less attention. Existing VLM bias studies often focus on portrait-style images and…
Decision MakingQuestion AnsweringVisual Question Answering (VQA)How well do NLI models capture verb veridicality?
In natural language inference (NLI), contexts are considered veridical if they allow us to infer that their underlying propositions make true claims about the real world. We investigate whether a state-of-the-art natural…
Natural Language InferenceNegationSentence