홈 › Papers

FSPO: Few-Shot Preference Optimization of Synthetic Preference Data in LLMs Elicits Effective Personalization to Real Users

2025-02-26 · Anikait Singh, Sheryl Hsu, Kyle Hsu, Eric Mitchell, Stefano Ermon, Tatsunori Hashimoto, Archit Sharma, Chelsea Finn

Effective personalization of LLMs is critical for a broad range of user-interfacing applications such as virtual assistants and content curation. Inspired by the strong in-context learning capabilities of LLMs, we propose Few-Shot Preference Optimization (FSPO), which reframes reward modeling as a meta-learning problem. Under this framework, an LLM learns to quickly adapt to a user via a few labeled preferences from that user, constructing a personalized reward function for them. Additionally, since real-world preference data is scarce and challenging to collect at scale, we propose careful design choices to construct synthetic preference datasets for personalization, generating over 1M synthetic personalized preferences using publicly available LLMs. In particular, to successfully transfer from synthetic data to real users, we find it crucial for the data to exhibit both high diversity and coherent, self-consistent structure. We evaluate FSPO on personalized open-ended generation for up to 1,500 synthetic users across across three domains: movie reviews, pedagogical adaptation based on educational background, and general question answering, along with a controlled human study. Overall, FSPO achieves an 87% Alpaca Eval winrate on average in generating responses that are personalized to synthetic users and a 72% winrate with real human users in open-ended question answering.

📄 PDF Abstract BibTeX arXiv:2502.19312

Code (1)

asap7772/fewshot-preference-optimization pytorch

Tasks

In-Context LearningMeta-LearningOpen-Ended Question AnsweringQuestion Answering

FsPONER: Few-shot Prompt Optimization for Named Entity Recognition in Domain-specific Scenarios

2024-07-10 · Yongjian Tang, Rakebul Hasan, Thomas Runkler

Large Language Models (LLMs) have provided a new pathway for Named Entity Recognition (NER) tasks. Compared with fine-tuning, LLM-powered prompting methods avoid the need for training, conserve substantial computational …

Few-Shot Learning named-entity-recognition Named Entity Recognition Named Entity Recognition (NER)+1

AIPO: Improving Training Objective for Iterative Preference Optimization

2024-09-13 · Yaojie Shen, Xinyao Wang, Yulei Niu, Ying Zhou 외

Preference Optimization (PO), is gaining popularity as an alternative choice of Proximal Policy Optimization (PPO) for aligning Large Language Models (LLMs). Recent research on aligning LLMs iteratively with synthetic or…

Group Preference Optimization: Few-Shot Alignment of Large Language Models

2023-10-17 · Siyan Zhao, John Dang, Aditya Grover

Many applications of large language models (LLMs), ranging from chatbots to creative writing, require nuanced subjective judgments that can differ significantly across different groups. Existing alignment algorithms can …

Few-Shot Learning Meta-Learning

GFRIEND: Generative Few-shot Reward Inference through EfficieNt DPO

2025-06-10 · Yiyang Zhao, Huiyu Bai, Xuejiao Zhao

The ability to train high-performing reward models with few-shot data is critical for enhancing the efficiency and scalability of Reinforcement Learning from Human Feedback (RLHF). We propose a data augmentation and expa…

Data Augmentation Model Optimization

Generative Reward Models

2024-10-02 · Dakota Mahan, Duy Van Phung, Rafael Rafailov, Chase Blagden 외

Reinforcement Learning from Human Feedback (RLHF) has greatly improved the performance of modern Large Language Models (LLMs). The RLHF process is resource-intensive and technically challenging, generally requiring a lar…

reinforcement-learning Reinforcement Learning