An Effectiveness Metric for Ordinal Classification: Formal Properties and Experimental Results
In Ordinal Classification tasks, items have to be assigned to classes that have a relative ordering, such as positive, neutral, negative in sentiment analysis. Remarkably, the most popular evaluation metrics for ordinal classification tasks either ignore relevant information (for instance, precision/recall on each of the classes ignores their relative ordering) or assume additional information (for instance, Mean Average Error assumes absolute distances between classes). In this paper we propose a new metric for Ordinal Classification, Closeness Evaluation Measure, that is rooted on Measurement Theory and Information Theory. Our theoretical analysis and experimental results over both synthetic data and data from NLP shared tasks indicate that the proposed metric captures quality aspects from different traditional tasks simultaneously. In addition, it generalizes some popular classification (nominal scale) and error minimization (interval scale) metrics, depending on the measurement scale in which it is instantiated.
Code (1)
Tasks
ClassificationGeneral ClassificationOrdinal ClassificationSentiment AnalysisSimilar Papers 제목 키워드 기반
Conformal Risk Control for Ordinal Classification
As a natural extension to the standard conformal prediction method, several conformal risk control methods have been recently developed and applied to various learning problems. In this work, we seek to control the confo…
ClassificationConformal PredictionDiabetic Retinopathy DetectionOrdinal ClassificationReliable Conformal Prediction for Ordinal Classification Using the Ranked Probability Score
Ordinal classification (OC) arises in high-stakes domains such as medicine and finance, where uncertainty quantification must account for the severity of ordinal errors. Conformal prediction (CP) provides distribution-fr…
Ordinal ClassificationA Novel Variational Autoencoder with Applications to Generative Modelling, Classification, and Ordinal Regression
We develop a novel probabilistic generative model based on the variational autoencoder approach. Notable aspects of our architecture are: a novel way of specifying the latent variables prior, and the introduction of an o…
ClassificationGeneral ClassificationOrdinal ClassificationregressionConformal Prediction Sets for Ordinal Classification
Ordinal classification (OC), i.e., labeling instances along classes with a natural ordering, is common in multiple applications such as size or budget based recommendations and disease severity labeling. Often in pract…
Distribution-free Conformal Prediction for Ordinal Classification
Conformal prediction is a general distribution-free approach for constructing prediction sets combined with any machine learning algorithm that achieve valid marginal or conditional coverage in finite samples. Ordinal cl…
ClassificationConformal PredictionOrdinal ClassificationPrediction+1