How Trustworthy are Performance Evaluations for Basic Vision Tasks?
This paper examines performance evaluation criteria for basic vision tasks involving sets of objects namely, object detection, instance-level segmentation and multi-object tracking. The rankings of algorithms by an existing criterion can fluctuate with different choices of parameters, e.g. Intersection over Union (IoU) threshold, making their evaluations unreliable. More importantly, there is no means to verify whether we can trust the evaluations of a criterion. This work suggests a notion of trustworthiness for performance criteria, which requires (i) robustness to parameters for reliability, (ii) contextual meaningfulness in sanity tests, and (iii) consistency with mathematical requirements such as the metric properties. We observe that these requirements were overlooked by many widely-used criteria, and explore alternative criteria using metrics for sets of shapes. We also assess all these criteria based on the suggested requirements for trustworthiness.
Code (1)
Tasks
Multi-Object Trackingobject-detectionObject DetectionObject TrackingSimilar Papers 제목 키워드 기반
A Trustworthy Framework for Medical Image Analysis with Deep Learning
Computer vision and machine learning are playing an increasingly important role in computer-assisted diagnosis; however, the application of deep learning to medical imaging has challenges in data availability and data im…
Deep LearningMedical Image AnalysisRevisiting Implicit Neural Representations in Low-Level Vision
Implicit Neural Representation (INR) has been emerging in computer vision in recent years. It has been shown to be effective in parameterising continuous signals such as dense 3D models from discrete image data, e.g. the…
DeblurringDenoisingImage DenoisingImage Restoration+2CONSISTENT: Open-Ended Question Generation From News Articles
Recent work on question generation has largely focused on factoid questions such as who, what, where, when about basic facts. Generating open-ended why, how, what, etc. questions that require long-form answers have prove…
ArticlesQuestion GenerationQuestion-GenerationTrustworthy Long-Tailed Classification
Classification on long-tailed distributed data is a challenging problem, which suffers from serious class-imbalance and accordingly unpromising performance especially on tail classes. Recently, the ensembling based metho…
ClassificationLong-tail LearningOut of Distribution (OOD) DetectionTowards Trustworthy Healthcare AI: Attention-Based Feature Learning for COVID-19 Screening With Chest Radiography
Building AI models with trustworthiness is important especially in regulated areas such as healthcare. In tackling COVID-19, previous work uses convolutional neural networks as the backbone architecture, which has shown …
Representation Learning