Common Sense Bias in Semantic Role Labeling
Large-scale language models such as ELMo and BERT have pushed the horizon of what is possible in semantic role labeling (SRL), solving the out-of-vocabulary problem and enabling end-to-end systems, but they have also introduced significant biases. We evaluate three SRL parsers on very simple transitive sentences with verbs usually associated with animate subjects and objects, such as, “Mary babysat Tom”: a state-of-the-art parser based on BERT, an older parser based on GloVe, and an even older parser from before the days of word embeddings. When arguments are word forms predominantly used as person names, aligning with common sense expectations of animacy, the BERT-based parser is unsurprisingly superior; yet, with abstract or random nouns, the opposite picture emerges. We refer to this as “common sense bias” and present a challenge dataset for evaluating the extent to which parsers are sensitive to such a bias. Our code and challenge dataset are available here: github.com/coastalcph/comte
Code (0)
등록된 구현이 없습니다.
Tasks
Common Sense ReasoningSemantic Role LabelingWord EmbeddingsSimilar Papers 제목 키워드 기반
Improving Implicit Semantic Role Labeling by Predicting Semantic Frame Arguments
Implicit semantic role labeling (iSRL) is the task of predicting the semantic roles of a predicate that do not appear as explicit arguments, but rather regard common sense knowledge or are mentioned earlier in the discou…
Common Sense ReasoningSemantic Role LabelingAn MRC Framework for Semantic Role Labeling
Semantic Role Labeling (SRL) aims at recognizing the predicate-argument structure of a sentence and can be decomposed into two subtasks: predicate disambiguation and argument labeling. Prior work deals with these two tas…
Computational EfficiencyMachine Reading ComprehensionMultiple-choiceReading Comprehension+2An MRC Framework for Semantic Role Labeling
Semantic Role Labeling (SRL) aims at recognizing the predicate-argument structure of a sentence and can be decomposed into two subtasks: predicate disambiguation and argument labeling. Prior work deals with these two tas…
Computational EfficiencyMachine Reading ComprehensionMultiple-choiceReading Comprehension+2InVeRo: Making Semantic Role Labeling Accessible with Intelligible Verbs and Roles
Semantic Role Labeling (SRL) is deeply dependent on complex linguistic resources and sophisticated neural models, which makes the task difficult to approach for non-experts. To address this issue we present a new platfor…
Semantic Role LabelingAffordance Extraction and Inference based on Semantic Role Labeling
Common-sense reasoning is becoming increasingly important for the advancement of Natural Language Processing. While word embeddings have been very successful, they cannot explain which aspects of 'coffee' and 'tea' make …
Common Sense ReasoningSemantic Role LabelingWord EmbeddingsWord Similarity