Prompt Optimization with Logged Bandit Data
We study how to use naturally available user feedback, such as clicks, to optimize large language model (LLM) pipelines for generating personalized sentences using prompts. Naive approaches, which estimate the policy gradient in the prompt space, suffer either from variance caused by the large action space of prompts or bias caused by inaccurate reward predictions. To circumvent these challenges, we propose a novel kernel-based off-policy gradient method, which estimates the policy gradient by leveraging similarity among generated sentences, substantially reducing variance while suppressing the bias. Empirical results on our newly established suite of benchmarks demonstrate the effectiveness of the proposed approach in generating personalized descriptions for movie recommendations, particularly when the number of candidate prompts is large.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModelingLanguage ModellingLarge Language ModelSimilar Papers 제목 키워드 기반
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
Counterfactual learning from human bandit feedback describes a scenario where user feedback on the quality of outputs of a historic system is logged and used to improve a target system. We show how to apply this learning…
counterfactualSemantic ParsingCounterfactual Risk Minimization: Learning from Logged Bandit Feedback
We develop a learning principle and an efficient algorithm for batch learning from logged bandit feedback. This learning setting is ubiquitous in online systems (e.g., ad placement, web search, recommendation), where an …
counterfactualMulti-Label ClassificationMUlTI-LABEL-ClASSIFICATIONDeep Learning with Logged Bandit Feedback
We propose a new output layer for deep neural networks that permits the use of logged contextual bandit feedback for training. Such contextual bandit feedback can be available in huge quantities (e.g., logs of search eng…
counterfactualDeep LearningObject RecognitionRecommendation SystemsOff-Policy Evaluation and Learning from Logged Bandit Feedback: Error Reduction via Surrogate Policy
When learning from a batch of logged bandit feedback, the discrepancy between the policy to be learned and the off-policy training data imposes statistical and computational challenges. Unlike classical supervised learni…
Multi-Label ClassificationMUlTI-LABEL-ClASSIFICATIONOff-policy evaluationRecommendation SystemsModel Inversion Networks for Model-Based Optimization
In this work, we aim to solve data-driven optimization problems, where the goal is to find an input that maximizes an unknown score function given access to a dataset of inputs with corresponding scores. When the inputs …
Bayesian Optimizationmodelvalid