Knowledge Gradient for Preference Learning
The knowledge gradient is a popular acquisition function in Bayesian optimization (BO) for optimizing black-box objectives with noisy function evaluations. Many practical settings, however, allow only pairwise comparison queries, yielding a preferential BO problem where direct function evaluations are unavailable. Extending the knowledge gradient to preferential BO is hindered by its computational challenge. At its core, the look-ahead step in the preferential setting requires computing a non-Gaussian posterior, which was previously considered intractable. In this paper, we address this challenge by deriving an exact and analytical knowledge gradient for preferential BO. We show that the exact knowledge gradient performs strongly on a suite of benchmark problems, often outperforming existing acquisition functions. In addition, we also present a case study illustrating the limitation of the knowledge gradient in certain scenarios.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
CATNIP: LLM Unlearning via Calibrated and Tokenized Negative Preference Alignment
Pretrained knowledge memorized in LLMs raises critical concerns over safety and privacy, which has motivated LLM Unlearning as a technique for selectively removing the influences of undesirable knowledge. Existing approa…
Preference Score Distillation: Leveraging 2D Rewards to Align Text-to-3D Generation with Human Preference
Human preference alignment presents a critical yet underexplored challenge for diffusion models in text-to-3D generation. Existing solutions typically require task-specific fine-tuning, posing significant hurdles in data…
3D GenerationBPO: Towards Balanced Preference Optimization between Knowledge Breadth and Depth in Alignment
Reinforcement Learning with Human Feedback (RLHF) is the key to the success of large language models (LLMs) in recent years. In this work, we first introduce the concepts of knowledge breadth and knowledge depth, which m…
InformativenessAuto-Rubric as Reward: From Implicit Preferences to Explicit Multimodal Generative Criteria
Aligning multimodal generative models with human preferences demands reward signals that respect the compositional, multi-dimensional structure of human judgment. Prevailing RLHF approaches reduce this structure to scala…
Text-to-Image GenerationImage EditingAligning Diffusion Language Models via Unpaired Preference Optimization
Diffusion language models (dLLMs) are an emerging alternative to autoregressive (AR) generators, but aligning them to human preferences is challenging because sequence log-likelihoods are intractable and pairwise prefere…