EntEval: A Holistic Evaluation Benchmark for Entity Representations

2019-08-31 · IJCNLP 2019 11 · Mingda Chen, Zewei Chu, Yang Chen, Karl Stratos, Kevin Gimpel

Rich entity representations are useful for a wide class of problems involving entities. Despite their importance, there is no standardized benchmark that evaluates the overall quality of entity representations. In this work, we propose EntEval: a test suite of diverse tasks that require nontrivial understanding of entities including entity typing, entity similarity, entity relation prediction, and entity disambiguation. In addition, we develop training techniques for learning better entity representations by using natural hyperlink annotations in Wikipedia. We identify effective objectives for incorporating the contextual information in hyperlinks into state-of-the-art pretrained language models and show that they improve strong baselines on multiple EntEval tasks.

📄 PDF Abstract BibTeX arXiv:1909.00137

Code (2)

ZeweiChu/EntEval 공식 구현 pytorch

mingdachen/bilm-tf 공식 구현 tf

Tasks

Entity Disambiguation Entity TypingRelation Prediction

SentEval: An Evaluation Toolkit for Universal Sentence Representations

2018-03-14 · LREC 2018 5 · Alexis Conneau, Douwe Kiela

We introduce SentEval, a toolkit for evaluating the quality of universal sentence representations. SentEval encompasses a variety of tasks, including binary and multi-class classification, natural language inference and …

General Classification Multi-class Classification Natural Language Inference Sentence+1

Is Language Modeling Enough? Evaluating Effective Embedding Combinations

2020-05-01 · LREC 2020 5 · Rudolf Schneider, Tom Oberhauser, Paul Grundmann, Felix Alex Gers 외

Universal embeddings, such as BERT or ELMo, are useful for a broad set of natural language processing tasks like text classification or sentiment analysis. Moreover, specialized embeddings also exist for tasks like topic…

Entity Disambiguation General Classification Language Modeling Language Modelling+5

Sentence Embeddings in NLI with Iterative Refinement Encoders

2018-08-27 · Aarne Talman, Anssi Yli-Jyrä, Jörg Tiedemann

Sentence-level representations are necessary for various NLP tasks. Recurrent neural networks have proven to be very effective in learning distributed representations and can be trained efficiently on natural language in…

Natural Language Inference Sentence Sentence Embedding Sentence-Embedding+2

StudentEval: A Benchmark of Student-Written Prompts for Large Language Models of Code

2023-06-07 · Hannah McLean Babe, Sydney Nguyen, Yangtian Zi, Arjun Guha 외

Code LLMs are being rapidly deployed and there is evidence that they can make professional programmers more productive. Current benchmarks for code generation measure whether models generate correct programs given an exp…

Code Generation

SufiSent - Universal Sentence Representations Using Suffix Encodings

2018-02-20 · Siddhartha Brahma

Computing universal distributed representations of sentences is a fundamental task in natural language processing. We propose a method to learn such representations by encoding the suffixes of word sequences in a sentenc…

Natural Language Inference Sentence