Investigating associative, switchable and negatable Winograd items on renewed French data sets
The Winograd Schema Challenge (WSC) consists of a set of anaphora resolution problems resolvable only by reasoning about world knowledge. This article describes the update of the existing French data set and the creation of three subsets allowing for a more robust, fine-grained evaluation protocol of WSC in French (FWSC) : an associative subset (items easily resolvable with lexical co-occurrence), a switchable subset (items where the inversion of two keywords reverses the answer) and a negatable subset (items where applying negation on its verb reverses the answer). Experiences on these data sets with CamemBERT reach SOTA performances. Our evaluation protocol showed in addition that the higher performance could be explained by the existence of associative items in FWSC. Besides, increasing the size of training corpus improves the model’s performance on switchable items while the impact of larger training corpus remains small on negatable items.
Code (1)
Tasks
NegationWorld KnowledgeSimilar Papers 제목 키워드 기반
Exploring Unsupervised Pretraining and Sentence Structure Modelling for Winograd Schema Challenge
Winograd Schema Challenge (WSC) was proposed as an AI-hard problem in testing computers' intelligence on common sense representation and reasoning. This paper presents the new state-of-theart on WSC, achieving an accurac…
Common Sense ReasoningSentenceThe Sensitivity of Language Models and Humans to Winograd Schema Perturbations
Large-scale pretrained language models are the major driving force behind recent improvements in performance on the Winograd Schema Challenge, a widely employed test of common sense reasoning ability. We show, however, w…
Common Sense ReasoningDiagnosticSensitivityMandarinograd: A Chinese Collection of Winograd Schemas
This article introduces Mandarinograd, a corpus of Winograd Schemas in Mandarin Chinese. Winograd Schemas are particularly challenging anaphora resolution problems, designed to involve common sense reasoning and to limit…
Common Sense ReasoningNatural Language InferenceNatural Language UnderstandingNeural Distributed Autoassociative Memories: A Survey
Introduction. Neural network models of autoassociative, distributed memory allow storage and retrieval of many items (vectors) where the number of stored items can exceed the vector dimension (the number of neurons in th…
RetrievalSurveyA novel HD Computing Algebra: Non-associative superposition of states creating sparse bundles representing order information
Information inflow into a computational system is by a sequence of information items. Cognitive computing, i.e. performing transformations along that sequence, requires to represent item information as well as sequential…
Navigate