Machine Learning Algorithms for Data Labeling: An Empirical Evaluation
The lack of labeled data is a major problem in both research and industrial settings since obtaining labels is often an expensive and time-consuming activity. In the past years, several machine learning algorithms were developed to assist and perform automated labeling in partially labeled datasets. While many of these algorithms are available in open-source packages, there is no research that investigates how these algorithms compare to each other in different types of datasets and with different percentages of available labels. To address this problem, this paper empirically evaluates and compares seven algorithms for automated labeling in terms of accuracy. We investigate how these algorithms perform in six different and well-known datasets with three different types of data, images, texts, and numerical values. We evaluate these algorithms under two different experimental conditions, with 10\% and 50\% labels of available labels in the dataset. Each algorithm, in each dataset for each experimental condition, is evaluated independently ten times with different random seeds. The results are analyzed and the algorithms are compared utilizing a Bayesian Bradley-Terry model. The results indicate that while the algorithms label spreading with K-nearest neighbors perform better in the aggregated results, the active learning algorithms query by instance QBC and query instance uncertainty sample perform better when there is only 10\% of labels available. These results can help machine learning practitioners in choosing optimal machine learning algorithms to label their data.
Code (0)
등록된 구현이 없습니다.
Tasks
Active LearningBIG-bench Machine LearningPartially Labeled DatasetsSimilar Papers 제목 키워드 기반
On Adversarial Bias and the Robustness of Fair Machine Learning
Optimizing prediction accuracy can come at the expense of fairness. Towards minimizing discrimination against a group, fair machine learning algorithms strive to equalize the behavior of a model across different groups, …
BIG-bench Machine LearningData PoisoningFairnessAn Empirical Evaluation of Sequence-Tagging Trainers
The task of assigning label sequences to a set of observed sequences is common in computational linguistics. Several models for sequence labeling have been proposed over the last few years. Here, we focus on discriminati…
Accelerated Labeling of Discrete Abstractions for Autonomous Driving Subject to LTL Specifications
Linear temporal logic and automaton-based run-time verification provide a powerful framework for designing task and motion planning algorithms for autonomous agents. The drawback to this approach is the computational cos…
Autonomous DrivingDecision MakingGPUMotion Planning+1Towards Comparable Active Learning
Active Learning has received significant attention in the field of machine learning for its potential in selecting the most informative samples for labeling, thereby reducing data annotation costs. However, we show that …
Active LearningAn Odor Labeling Convolutional Encoder-Decoder for Odor Sensing in Machine Olfaction
Deep learning methods have been widely applied to visual and acoustic technology. In this paper, we proposed an odor labeling convolutional encoder-decoder (OLCE) for odor identification in machine olfaction. OLCE compos…
Decoder