Soft Prompts for Evaluation: Measuring Conditional Distance of Capabilities
To help evaluate and understand the latent capabilities of language models, this paper introduces an approach using optimized input embeddings, or 'soft prompts,' as a metric of conditional distance between a model and a target behavior. The technique aims to facilitate latent capability discovery as a part of automated red teaming/evaluation suites and to provide quantitative feedback about the accessibility of potentially concerning behaviors in a way that may scale to powerful future models, including those which may otherwise be capable of deceptive alignment. An evaluation framework using soft prompts is demonstrated in natural language, chess, and pathfinding, and the technique is extended with generalized conditional soft prompts to aid in constructing task evaluations.
Code (1)
Tasks
Red TeamingSimilar Papers 제목 키워드 기반
On Distance and Kernel Measures of Conditional Independence
Measuring conditional independence is one of the important tasks in statistical inference and is fundamental in causal discovery, feature selection, dimensionality reduction, Bayesian network learning, and others. In thi…
Causal DiscoveryDimensionality Reductionfeature selectionMeasuring Policy Distance for Multi-Agent Reinforcement Learning
Diversity plays a crucial role in improving the performance of multi-agent reinforcement learning (MARL). Currently, many diversity-based methods have been developed to overcome the drawbacks of excessive parameter shari…
DiversityMulti-agent Reinforcement Learningreinforcement-learningReinforcement LearningSCP: Soft Conditional Prompt Learning for Aerial Video Action Recognition
We present a new learning approach, Soft Conditional Prompt Learning (SCP), which leverages the strengths of prompt learning for aerial video action recognition. Our approach is designed to predict the action of each age…
Action RecognitionOptical Flow EstimationPrompt LearningTemporal Action LocalizationWords Worth a Thousand Pictures: Measuring and Understanding Perceptual Variability in Text-to-Image Generation
Diffusion models are the state of the art in text-to-image generation, but their perceptual variability remains understudied. In this paper, we examine how prompts affect image variability in black-box diffusion-based mo…
Image GenerationPerceptual DistanceText to Image GenerationText-to-Image GenerationLearning Deep Classifiers Consistent With Fine-Grained Novelty Detection
The problem of novelty detection in fine-grained visual classification (FGVC) is considered. An integrated understanding of the probabilistic and distance-based approaches to novelty detection is developed within the…
Fine-Grained Image ClassificationNovelty Detection