Was that so hard? Estimating human classification difficulty
When doctors are trained to diagnose a specific disease, they learn faster when presented with cases in order of increasing difficulty. This creates the need for automatically estimating how difficult it is for doctors to classify a given case. In this paper, we introduce methods for estimating how hard it is for a doctor to diagnose a case represented by a medical image, both when ground truth difficulties are available for training, and when they are not. Our methods are based on embeddings obtained with deep metric learning. Additionally, we introduce a practical method for obtaining ground truth human difficulty for each image case in a dataset using self-assessed certainty. We apply our methods to two different medical datasets, achieving high Kendall rank correlation coefficients, showing that we outperform existing methods by a large margin on our problem and data.
Code (0)
등록된 구현이 없습니다.
Tasks
ClassificationMetric LearningSimilar Papers 제목 키워드 기반
How hard can it be? Estimating the difficulty of visual search in an image
We address the problem of estimating image difficulty defined as the human response time for solving a visual search task. We collect human annotations of image difficulty for the PASCAL VOC 2012 data set through a crowd…
Object LocalizationWeakly-Supervised Object LocalizationHuman not in the loop: objective sample difficulty measures for Curriculum Learning
Curriculum learning is a learning method that trains models in a meaningful order from easier to harder samples. A key here is to devise automatic and objective difficulty measures of samples. In the medical domain, prev…
ClassificationText Difficulty Study: Do machines behave the same as humans regarding text difficulty?
Given a task, human learns from easy to hard, whereas the model learns randomly. Undeniably, difficulty insensitive learning leads to great success in NLP, but little attention has been paid to the effect of text difficu…
Transferability and Hardness of Supervised Classification Tasks
We propose a novel approach for estimating the difficulty and transferability of supervised classification tasks. Unlike previous work, our approach is solution agnostic and does not require or assume trained models. Ins…
AttributeClassificationFace RecognitionGeneral ClassificationDifficulty in estimating visual information from randomly sampled images
In this paper, we evaluate dimensionality reduction methods in terms of difficulty in estimating visual information on original images from dimensionally reduced ones. Recently, dimensionality reduction has been receivin…
BIG-bench Machine LearningDimensionality Reductionimage-classificationImage Classification+1