NLPStatTest: A Toolkit for Comparing NLP System Performance
Statistical significance testing centered on p-values is commonly used to compare NLP system performance, but p-values alone are insufficient because statistical significance differs from practical significance. The latter can be measured by estimating effect size. In this paper, we propose a three-stage procedure for comparing NLP system performance and provide a toolkit, NLPStatTest, that automates the process. Users can upload NLP system evaluation scores and the toolkit will analyze these scores, run appropriate significance tests, estimate effect size, and conduct power analysis to estimate Type II error. The toolkit provides a convenient and systematic way to compare NLP system performance that goes beyond statistical significance testing
Code (1)
Similar Papers 제목 키워드 기반
mwetoolkit+sem: Integrating Word Embeddings in the mwetoolkit for Semantic MWE Processing
This paper presents mwetoolkit+sem: an extension of the mwetoolkit that estimates semantic compositionality scores for multiword expressions (MWEs) based on word embeddings. First, we describe our implementation of vecto…
Word EmbeddingsMatchZoo: A Toolkit for Deep Text Matching
In recent years, deep neural models have been widely adopted for text matching tasks, such as question answering and information retrieval, showing improved performance as compared with previous methods. In this paper, w…
Ad-Hoc Information RetrievalInformation RetrievalQuestion AnsweringRetrieval+1A Review, Framework and R toolkit for Exploring, Evaluating, and Comparing Visualizations
This paper gives a review and synthesis of methods of evaluating dimensionality reduction techniques. Particular attention is paid to rank-order neighborhood evaluation metrics. A framework is created for exploring dimen…
Dimensionality ReductionQueryGym: A Toolkit for Reproducible LLM-Based Query Reformulation
We present QueryGym, a lightweight, extensible Python toolkit that supports large language model (LLM)-based query reformulation. This is an important tool development since recent work on llm-based query reformulation h…
FlashRAG: A Modular Toolkit for Efficient Retrieval-Augmented Generation Research
With the advent of large language models (LLMs) and multimodal large language models (MLLMs), the potential of retrieval-augmented generation (RAG) has attracted considerable research attention. Various novel algorithms …
RAGRetrievalRetrieval-augmented Generation