Complementarity, F-score, and NLP Evaluation
This paper addresses the problem of quantifying the differences between entity extraction systems, where in general only a small proportion a document should be selected. Comparing overall accuracy is not very useful in these cases, as small differences in accuracy may correspond to huge differences in selections over the target minority class. Conventionally, one may use per-token complementarity to describe these differences, but it is not very useful when the set is heavily skewed. In such situations, which are common in information retrieval and entity recognition, metrics like precision and recall are typically used to describe performance. However, precision and recall fail to describe the differences between sets of objects selected by different decision strategies, instead just describing the proportional amount of correct and incorrect objects selected. This paper presents a method for measuring complementarity for precision, recall and F-score, quantifying the difference between entity extraction approaches.
Code (0)
등록된 구현이 없습니다.
Tasks
Entity Extraction using GANInformation RetrievalRetrievalSimilar Papers 제목 키워드 기반
Better Together: Evaluating the Complementarity of Earth Embedding Models
Earth embedding models transform Earth observation data into embeddings uniquely tied to locations on the Earth's surface. These models are typically evaluated in isolation, comparing the downstream task performance acro…
Self-supervision and Learnable STRFs for Age, Emotion, and Country Prediction
This work presents a multitask approach to the simultaneous estimation of age, country of origin, and emotion given vocal burst audio for the 2022 ICML Expressive Vocalizations Challenge ExVo-MultiTask track. The method …
DecoderShape Complementarity Analysis for Objects of Arbitrary Shape
The basic problem of shape complementarity analysis appears fundamental to applications as diverse as mechanical design, assembly automation, robot motion planning, micro- and nano-fabrication, protein-ligand binding, an…
Drug Designglobal-optimizationMotion PlanningExploring Complementarity and Explainability in CNNs for Periocular Verification Across Acquisition Distances
We study the complementarity of different CNNs for periocular verification at different distances on the UBIPr database. We train three architectures of increasing complexity (SqueezeNet, MobileNetv2, and ResNet50) on a …
Explicit View-labels Matter: A Multifacet Complementarity Study of Multi-view Clustering
Consistency and complementarity are two key ingredients for boosting multi-view clustering (MVC). Recently with the introduction of popular contrastive learning, the consistency learning of views has been further enhance…
ClusteringContrastive LearningDecoder