Does it Really Generalize Well on Unseen Data? Systematic Evaluation of Relational Triple Extraction Methods
The ability to extract entities and their relations from unstructured text is essential for the automated maintenance of large-scale knowledge graphs. To keep a knowledge graph up-to-date, an extractor needs not only the ability to recall the triples it encountered during training, but also the ability to extract the new triples from the context that it has never seen before. In this paper, we show that although existing extraction models are able to easily memorize and recall already seen triples, they cannot generalize effectively for unseen triples. This alarming observation was previously unknown due to the composition of the test sets of the go-to benchmark datasets, which turns out to contain only 2% unseen data, rendering them incapable to measure the generalization performance. To separately measure the generalization performance from the memorization performance, we emphasize unseen data by rearranging datasets, sifting out training instances, or augmenting test sets. In addition to that, we present a simple yet effective augmentation technique to promote generalization of existing extraction models, and experimentally confirm that the proposed method can significantly increase the generalization performance of existing models.
Code (0)
등록된 구현이 없습니다.
Tasks
Knowledge GraphsMemorizationSimilar Papers 제목 키워드 기반
Towards Generalizeable Semantic Product Search by Text Similarity Pre-training on Search Click Logs
Recently, semantic search has been successfully applied to E-commerce product search and the learned semantic space for query and product encoding are expected to generalize well to unseen queries or products. Yet, wheth…
text similarityDoes BERT really agree ? Fine-grained Analysis of Lexical Dependence on a Syntactic Task
Although transformer-based Neural Language Models obtain impressive results on a wide variety of tasks, their generalization abilities are not well understood. They have been shown to perform strongly on subject-verb num…
Does BERT really agree ? Fine-grained Analysis of Lexical Dependence on a Syntactic Task
Although transformer-based Neural Language Models demonstrate impressive performance on a variety of tasks, their generalization abilities are not well understood. They have been shown to perform strongly on subject-verb…
Learning shared manifold representation of images and attributes for generalized zero-shot learning
Many of the zero-shot learning methods have realized predicting labels of unseen images by learning the relations between images and pre-defined class-attributes. However, recent studies show that, under the more realist…
Generalized Zero-Shot LearningZero-Shot LearningDoes GPT Really Get It? A Hierarchical Scale to Quantify Human vs AI's Understanding of Algorithms
As Large Language Models (LLMs) perform (and sometimes excel at) more and more complex cognitive tasks, a natural question is whether AI really understands. The study of understanding in LLMs is in its infancy, and the c…
Philosophy