Conformal Predictor for Improving Zero-shot Text Classification Efficiency
Pre-trained language models (PLMs) have been shown effective for zero-shot (0shot) text classification. 0shot models based on natural language inference (NLI) and next sentence prediction (NSP) employ cross-encoder architecture and infer by making a forward pass through the model for each label-text pair separately. This increases the computational cost to make inferences linearly in the number of labels. In this work, we improve the efficiency of such cross-encoder-based 0shot models by restricting the number of likely labels using another fast base classifier-based conformal predictor (CP) calibrated on samples labeled by the 0shot model. Since a CP generates prediction sets with coverage guarantees, it reduces the number of target labels without excluding the most probable label based on the 0shot model. We experiment with three intent and two topic classification datasets. With a suitable CP for each dataset, we reduce the average inference time for NLI- and NSP-based models by 25.6% and 22.2% respectively, without dropping performance below the predefined error rate of 1%.
Code (0)
등록된 구현이 없습니다.
Tasks
ClassificationNatural Language InferenceSentencetext-classificationText ClassificationTopic ClassificationZero-Shot Text ClassificationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
CAOS: Conformal Aggregation of One-Shot Predictors
One-shot prediction enables rapid adaptation of pretrained foundation models to new tasks using only one labeled example, but lacks principled uncertainty quantification. While conformal prediction provides finite-sample…
Text ClassificationTowards Reliable Zero Shot Classification in Self-Supervised Models with Conformal Prediction
Self-supervised models trained with a contrastive loss such as CLIP have shown to be very powerful in zero-shot classification settings. However, to be used as a zero-shot classifier these models require the user to prov…
ClassificationConformal PredictionOutlier Detectionzero-shot-classification+1Sparse Activations as Conformal Predictors
Conformal prediction is a distribution-free framework for uncertainty quantification that replaces point predictions with sets, offering marginal coverage guarantees (i.e., ensuring that the prediction sets contain the t…
Conformal PredictionPredictiontext-classificationText Classification+1Optimized conformal classification using gradient descent approximation
Conformal predictors are an important class of algorithms that allow predictions to be made with a user-defined confidence level. They are able to do this by outputting prediction sets, rather than simple point predictio…
BIG-bench Machine LearningClassificationvalidAggregation in conformal e-classification
Aggregating conformal predictors is a standard way of balancing their predictive and computational efficiency while retaining their validity, at least approximately. An important advantage of conformal e-predictors is th…
Computational Efficiency