Incorporating Subjectivity into Gendered Ambiguous Pronoun (GAP) Resolution using Style Transfer
The GAP dataset is a Wikipedia-based evaluation dataset for gender bias detection in coreference resolution, containing mostly objective sentences. Since subjectivity is ubiquitous in our daily texts, it becomes necessary to evaluate models for both subjective and objective instances. In this work, we present a new evaluation dataset for gender bias in coreference resolution, GAP-Subjective, which increases the coverage of the original GAP dataset by including subjective sentences. We outline the methodology used to create this dataset. Firstly, we detect objective sentences and transfer them into their subjective variants using a sequence-to-sequence model. Secondly, we outline the thresholding techniques based on fluency and content preservation to maintain the quality of the sentences. Thirdly, we perform automated and human-based analysis of the style transfer and infer that the transferred sentences are of high quality. Finally, we benchmark both GAP and GAP-Subjective datasets using a BERT-based model and analyze its predictive performance and gender bias.
Code (0)
등록된 구현이 없습니다.
Tasks
Bias Detectioncoreference-resolutionCoreference ResolutionGender Bias DetectionStyle TransferSimilar Papers 제목 키워드 기반
Gendered Ambiguous Pronouns Shared Task: Boosting Model Confidence by Evidence Pooling
This paper presents a strong set of results for resolving gendered ambiguous pronouns on the Gendered Ambiguous Pronouns shared task. The model presented here draws upon the strengths of state-of-the-art language and cor…
coreference-resolutionCoreference ResolutionGendered Ambiguous Pronouns Shared Task: Boosting Model Confidence by Evidence Pooling
This paper presents a strong set of results for resolving gendered ambiguous pronouns on the Gendered Ambiguous Pronouns shared task. The model presented here draws upon the strengths of state-of-the-art language and cor…
coreference-resolutionCoreference ResolutionGendered Ambiguous Pronoun (GAP) Shared Task at the Gender Bias in NLP Workshop 2019
The 1st ACL workshop on Gender Bias in Natural Language Processing included a shared task on gendered ambiguous pronoun (GAP) resolution. This task was based on the coreference challenge defined in Webster et al. (2018),…
Mind the GAP: A Balanced Corpus of Gendered Ambiguous Pronouns
Coreference resolution is an important task for natural language understanding, and the resolution of ambiguous pronouns a longstanding challenge. Nonetheless, existing corpora do not capture ambiguous pronouns in suffic…
Coreference ResolutionDiversityNatural Language UnderstandingType B Reflexivization as an Unambiguous Testbed for Multilingual Multi-Task Gender Bias
The one-sided focus on English in previous studies of gender bias in NLP misses out on opportunities in other languages: English challenge datasets such as GAP and WinoGender highlight model preferences that are "halluci…
Translation