Cascade Reward Sampling for Efficient Decoding-Time Alignment

2024-06-24 · Bolian Li, Yifan Wang, Anamika Lochab, Ananth Grama, Ruqi Zhang

Aligning large language models (LLMs) with human preferences is essential for their applications. Recently, decoding-time alignment has emerged as an effective plug-and-play technique that avoids fine-tuning model parameters. This approach retains the general utility of pretrained LLMs but often suffers from significant inefficiencies during decoding, primarily due to wasted token generation and excessive reward evaluations. To address these challenges, we introduce Cascade Reward Sampling (CARDS) to resolve both efficiency bottlenecks in decoding-time alignment. Specifically, we develop a segment-level rejection sampling algorithm that minimizes redundant computations of both LLMs and reward models (RMs). Central to CARDS is an uncertainty-based segmentation mechanism, which ensures the accuracy of RMs evaluations on incomplete segments. Furthermore, we provide a detailed analysis of reward scores on segments to elucidate the improved alignment performance. Experimental results demonstrate that CARDS significantly improves decoding efficiency, alignment quality, and general utility compared to existing decoding-time alignment methods, achieving approximately a 70% reduction in decoding time and over 90% win-ties in utility and safety benchmarks.

📄 PDF Abstract BibTeX arXiv:2406.16306

Code (1)

lblaoke/CARDS 공식 구현 pytorch

Tasks

Text Generation

InfAlign: Inference-aware language model alignment

2024-12-27 · Ananth Balashankar, Ziteng Sun, Jonathan Berant, Jacob Eisenstein 외

Language model alignment is a critical step in training modern generative language models. Alignment targets to improve win rate of a sample from the aligned model against the base model. Today, we are increasingly using…

Language Modeling Language Modelling model Models Alignment

Safe Inference-Time Alignment via Lagrangian Reward Augmentation

2026-07-02 · Yaswanth Chittepu, Ativ Joshi, Sohini Chintala, Scott Niekum arxiv

Inference-time alignment steers a frozen language model during decoding using auxiliary reward signals, avoiding the cost of repeated weight updates. However, existing inference-time alignment methods typically optimize …

Gradient-Guided Reward Optimization for Inference-time Alignment

2026-06-08 · Hankun Lin, Ruqi Zhang arxiv

Ensuring the reliability of Large Language Models (LLMs) under distribution drift requires inference-time adaptation. While inference-time alignment methods such as Best-of-$N$ and rejection sampling are widely used, the…

Controlled Decoding from Language Models

2023-10-25 · Sidharth Mudgal, Jong Lee, Harish Ganapathy, Yaguang Li 외

KL-regularized reinforcement learning (RL) is a popular alignment framework to control the language model responses towards high reward outcomes. We pose a tokenwise RL objective and propose a modular solver for it, call…

Language Modeling Language Modelling Multi-Objective Reinforcement Learning reinforcement-learning+2

Robust Multi-Objective Controlled Decoding of Large Language Models

2025-03-11 · Seongho Son, William Bankes, Sangwoong Yoon, Shyam Sundhar Ramesh 외

Test-time alignment of Large Language Models (LLMs) to human preferences offers a flexible way to generate responses aligned to diverse objectives without extensive retraining of LLMs. Existing methods achieve alignment …

Instruction Following