Active Testing: An Unbiased Evaluation Method for Distantly Supervised Relation Extraction
Distant supervision has been a widely used method for neural relation extraction for its convenience of automatically labeling datasets. However, existing works on distantly supervised relation extraction suffer from the low quality of test set, which leads to considerable biased performance evaluation. These biases not only result in unfair evaluations but also mislead the optimization of neural relation extraction. To mitigate this problem, we propose a novel evaluation method named active testing through utilizing both the noisy test set and a few manual annotations. Experiments on a widely used benchmark show that our proposed approach can yield approximately unbiased evaluations for distantly supervised relation extractors.
Code (0)
등록된 구현이 없습니다.
Tasks
RelationRelation ExtractionSimilar Papers 제목 키워드 기반
Towards Accurate and Consistent Evaluation: A Dataset for Distantly-Supervised Relation Extraction
In recent years, distantly-supervised relation extraction has achieved a certain success by using deep neural networks. Distant Supervision (DS) can automatically generate large-scale annotated data by aligning entity pa…
RelationRelation ExtractionBootstrapping Distantly Supervised IE using Joint Learning and Small Well-structured Corpora
We propose a framework to improve performance of distantly-supervised relation extraction, by jointly learning to solve two related tasks: concept-instance extraction and relation extraction. We combine this with a novel…
RelationRelation ExtractionConcept Pointer Network for Abstractive Summarization
A quality abstractive summary should not only copy salient source texts as summaries but should also tend to generate new conceptual words to express concrete details. Inspired by the popular pointer generator sequence-t…
Abstractive Text SummarizationText SummarizationActively Testing Your Model While It Learns: Realizing Label-Efficient Learning in Practice
In active learning (AL), we focus on reducing the data annotation cost from the model training perspective. However, "testing'', which often refers to the model evaluation process of using empirical risk to estimate the …