Prompt-Induced Score Variance in Zero-Shot Binary Vision-Language Safety Classification
Single-prompt first-token probabilities from zero-shot vision-language model (VLM) safety classifiers are treated as decision scores, but we show they are unreliable under semantically equivalent prompt reformulation: even when the binary label is constrained to a fixed output position, equivalent prompts can induce materially different unsafe probabilities for the same sample. Across multimodal safety benchmarks and multiple VLM families, cross-prompt variance is strongly associated with prompt-level disagreement and higher error, making it a useful fragility diagnostic. A training-free mean ensemble improves NLL on all 14 dataset-model evaluation pairs and ECE on 12/14 relative to a train-selected single-prompt baseline, and wins more head-to-head NLL comparisons than labeled temperature scaling, Platt scaling, and isotonic regression applied to the same prompt. Ranking gains are consistent against the train-selected baseline on both AUROC and AUPRC, and against the full 15-prompt distribution remain consistent on AUPRC while softening on AUROC. Labeled calibration on top of the mean provides further gains when labels are available, identifying prompt averaging as a strong label-free first stage rather than a replacement for calibration. We frame this as a reliability stress test for zero-shot VLM first-token safety scores and recommend prompt-family evaluation with mean aggregation as a standard label-free reliability baseline.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
An Exploration of Prompt-Based Zero-Shot Relation Extraction Method
Zero-shot relation extraction is an important method for dealing with the newly emerging relations in the real world which lacks labeled data. However, the current zero-shot methods usually rely on large-scale and in-dom…
Language ModelingLanguage ModellingRelationRelation ExtractionImproving Zero-shot Visual Question Answering via Large Language Models with Reasoning Question Prompts
Zero-shot Visual Question Answering (VQA) is a prominent vision-language task that examines both the visual and textual understanding capability of systems in the absence of training data. Recently, by converting the ima…
Question AnsweringSentenceVisual Question AnsweringVisual Question Answering (VQA)+1SPARC: Score Prompting and Adaptive Fusion for Zero-Shot Multi-Label Recognition in Vision-Language Models
Zero-shot multi-label recognition (MLR) with Vision-Language Models (VLMs) faces significant challenges without training data, model tuning, or architectural modifications. Existing approaches require prompt tuning or ar…
Language ModelingLanguage ModellingLarge Language ModelObjectOnce a Response, Always a Response: Detecting LLM-generated Text via Latent Prompt Restoration
Large language models (LLMs) can generate fluent and convincing text at scale, creating growing risks for misinformation dissemination, educational misuse, and platform governance. These concerns make robust detection of…
Multitask Prompted Training Enables Zero-Shot Task Generalization
Large language models have recently been shown to attain reasonable zero-shot generalization on a diverse set of tasks (Brown et al., 2020). It has been hypothesized that this is a consequence of implicit multitask learn…
BenchmarkingDecoderLanguage ModellingPrompt Engineering+1