Submodular Evaluation Subset Selection in Automatic Prompt Optimization
Automatic prompt optimization reduces manual prompt engineering, but relies on task performance measured on a small, often randomly sampled evaluation subset as its main source of feedback signal. Despite this, how to select that evaluation subset is usually treated as an implementation detail. We study evaluation subset selection for prompt optimization from a principled perspective and propose SESS, a submodular evaluation subset selection method. We frame selection as maximizing an objective set function and show that, under mild conditions, it is monotone and submodular, enabling greedy selection with theoretical guarantees. Across GSM8K, MATH, and GPQA-Diamond, submodularly selected evaluation subsets can yield better optimized prompts than random or heuristic baselines.
Code (0)
등록된 구현이 없습니다.
Tasks
Prompt EngineeringSimilar Papers 제목 키워드 기반
Sub-SA: Strengthen In-context Learning via Submodular Selective Annotation
In-context learning (ICL) leverages in-context examples as prompts for the predictions of Large Language Models (LLMs). These prompts play a crucial role in achieving strong performance. However, the selection of suitabl…
DiversityIn-Context LearningSelect Smarter, Not More: Prompt-Aware Evaluation Scheduling with Submodular Guarantees
Automatic prompt optimization (APO) hinges on the quality of its evaluation signal, yet scoring every prompt candidate on the full training set is prohibitively expensive. Existing methods either fix a single evaluation …
Causal meets Submodular: Subset Selection with Directed Information
We study causal subset selection with Directed Information as the measure of prediction causality. Two typical tasks, causal sensor placement and covariate selection, are correspondingly formulated into cardinality const…
Multinoulli Extension: A Lossless Continuous Relaxation for Partition-Constrained Subset Selection
Identifying the most representative subset for a close-to-submodular objective while satisfying the predefined partition constraint is a fundamental task with numerous applications in machine learning. However, the exist…
Maximizing Submodular Functions for Recommendation in the Presence of Biases
Subset selection tasks, arise in recommendation systems and search engines and ask to select a subset of items that maximize the value for the user. The values of subsets often display diminishing returns, and hence, sub…
FairnessRecommendation Systems