Evaluation of Coreference Resolution Systems Under Adversarial Attacks
A substantial overlap of coreferent mentions in the CoNLL dataset magnifies the recent progress on coreference resolution. This is because the CoNLL benchmark fails to evaluate the ability of coreference resolvers that requires linking novel mentions unseen at train time. In this work, we create a new dataset based on CoNLL, which largely decreases mention overlaps in the entire dataset and exposes the limitations of published resolvers on two aspects—lexical inference ability and understanding of low-level orthographic noise. Our findings show (1) the requirements for embeddings, used in resolvers, and for coreference resolutions are, by design, in conflict and (2) adversarial approaches are sometimes not legitimate to mitigate the obstacles, as they may falsely introduce mention overlaps in adversarial training and test sets, thus giving an inflated impression for the improvements.
Code (0)
등록된 구현이 없습니다.
Tasks
coreference-resolutionCoreference ResolutionSimilar Papers 제목 키워드 기반
Evaluation of named entity coreference
In many NLP applications like search and information extraction for named entities, it is necessary to find all the mentions of a named entity, some of which appear as pronouns (she, his, etc.) or nominals (the professor…
coreference-resolutionCoreference ResolutionEvaluating Coreference Resolvers on Community-based Question Answering: From Rule-based to State of the Art
Coreference resolution is a key step in natural language understanding. Developments in coreference resolution are mainly focused on improving the performance on standard datasets annotated for coreference resolution. Ho…
Answer Selectioncoreference-resolutionCoreference ResolutionNatural Language Understanding+1Named Person Coreference in English News
People are often entities of interest in tasks such as search and information extraction. In these tasks, the goal is to find as much information as possible about people specified by their name. However in text, some of…
coreference-resolutionCoreference Resolutionnamed-entity-recognitionNamed Entity Recognition+1Improving Generalization in Coreference Resolution via Adversarial Training
In order for coreference resolution systems to be useful in practice, they must be able to generalize to new text. In this work, we demonstrate that the performance of the state-of-the-art system decreases when the names…
coreference-resolutionCoreference ResolutionFindings of the Shared Task on Multilingual Coreference Resolution
This paper presents an overview of the shared task on multilingual coreference resolution associated with the CRAC 2022 workshop. Shared task participants were supposed to develop trainable systems capable of identifying…
Clusteringcoreference-resolutionCoreference Resolution