Using tournaments to calculate AUROC for zero-shot classification with LLMs
Large language models perform surprisingly well on many zero-shot classification tasks, but are difficult to fairly compare to supervised classifiers due to the lack of a modifiable decision boundary. In this work, we propose and evaluate a method that converts binary classification tasks into pairwise comparison tasks, obtaining relative rankings from LLMs. Repeated pairwise comparisons can be used to score instances using the Elo rating system (used in chess and other competitions), inducing a confidence ordering over instances in a dataset. We evaluate scheduling algorithms for their ability to minimize comparisons, and show that our proposed algorithm leads to improved classification performance, while also providing more information than traditional zero-shot classification.
Code (0)
등록된 구현이 없습니다.
Tasks
Binary ClassificationClassificationSchedulingzero-shot-classificationZero-Shot LearningSimilar Papers 제목 키워드 기반
WinCLIP: Zero-/Few-Shot Anomaly Classification and Segmentation
Visual anomaly classification and segmentation are vital for automating industrial quality inspection. The focus of prior research in the field has been on training custom models for each quality inspection task, which r…
Anomaly ClassificationAnomaly DetectionClassificationLanguage Modeling+3Exploring the Versatility of Zero-Shot CLIP for Interstitial Lung Disease Classification
Interstitial lung diseases (ILD) present diagnostic challenges due to their varied manifestations and overlapping imaging features. To address this, we propose a machine learning approach that utilizes CLIP, a multimodal…
ClassificationDiagnosticimage-classificationImage Classification+2Embedding-Driven Diversity Sampling to Improve Few-Shot Synthetic Data Generation
Accurate classification of clinical text often requires fine-tuning pre-trained language models, a process that is costly and time-consuming due to the need for high-quality data and expert annotators. Synthetic data gen…
DiversitySynthetic Data GenerationSignificantly improving zero-shot X-ray pathology classification via fine-tuning pre-trained image-text encoders
Deep neural networks are increasingly used in medical imaging for tasks such as pathological classification, but they face challenges due to the scarcity of high-quality, expert-labeled training data. Recent efforts have…
ClassificationContrastive LearningSentencezero-shot-classification+1Latent Sculpting for Zero-Shot Generalization: A Manifold Learning Approach to Out-of-Distribution Anomaly Detection
A critical vulnerability of supervised deep learning in high-dimensional tabular domains is "generalization collapse": models form precise decision boundaries around known training distributions but fail catastrophically…
Zero-shot GeneralizationRepresentation LearningDensity EstimationAnomaly Detection