Learning a Canonical Basis of Human Preferences from Binary Ratings

2025-03-31 · Kailas Vodrahalli, Wei Wei, James Zou

Recent advances in generative AI have been driven by alignment techniques such as reinforcement learning from human feedback (RLHF). RLHF and related techniques typically involve constructing a dataset of binary or ranked choice human preferences and subsequently fine-tuning models to align with these preferences. This paper shifts the focus to understanding the preferences encoded in such datasets and identifying common human preferences. We find that a small subset of 21 preference categories (selected from a set of nearly 5,000 distinct preferences) captures >89% of preference variation across individuals. This small set of preferences is analogous to a canonical basis of human preferences, similar to established findings that characterize human variation in psychology or facial recognition studies. Through both synthetic and empirical evaluations, we confirm that our low-rank, canonical set of human preferences generalizes across the entire dataset and within specific topics. We further demonstrate our preference basis' utility in model evaluation, where our preference categories offer deeper insights into model alignment, and in model training, where we show that fine-tuning on preference-defined subsets successfully aligns the model accordingly.

📄 PDF Abstract BibTeX arXiv:2503.24150

Code (0)

등록된 구현이 없습니다.

Methods 이 논문이 사용한 방법론

SET Dynamic Sparse Training method where weight mask is updated randomly periodically

ALIGN In the ALIGN method, visual and language representations are jointly trained from noisy image alt-text data. The image and text encoders are learned via contrastive loss…

Focus 설명 없음

Adaptive Margin RLHF via Preference over Preferences

2025-09-26 · Yaswanth Chittepu, Prasann Singhal, Greg Durrett, Scott Niekum arxiv

Margin-based optimization is fundamental to improving generalization and robustness in classification tasks. In the context of reward model learning from preferences within Reinforcement Learning from Human Feedback (RLH…

Reinforcement Learning

Reward Learning through Ranking Mean Squared Error

2026-01-14 · Chaitanya Kharyal, Calarina Muslimani, Matthew E. Taylor arxiv

Reward design remains a significant bottleneck in applying reinforcement learning (RL) to real-world problems. A popular alternative is reward learning, where reward functions are inferred from human feedback rather than…

Reinforcement Learning OpenAI Gym

MM-SCALE: Grounded Multimodal Moral Reasoning via Scalar Judgment and Listwise Alignment

2026-02-03 · Eunkyu Park, Wesley Hanwen Deng, Cheyon Jin, Matheus Kunzler Maldaner 외 arxiv

Vision-Language Models (VLMs) continue to struggle to make morally salient judgments in multimodal and socially ambiguous contexts. Prior works typically rely on binary or pairwise supervision, which often fail to captur…

Rating-based Reinforcement Learning

2023-07-30 · Devin White, Mingkang Wu, Ellen Novoseller, Vernon J. Lawhern 외

This paper develops a novel rating-based reinforcement learning approach that uses human ratings to obtain human guidance in reinforcement learning. Different from the existing preference-based and ranking-based reinforc…

reinforcement-learning Reinforcement Learning

Rich Insights from Cheap Signals: Efficient Evaluations via Tensor Factorization

2026-03-02 · Felipe Maia Polo, Aida Nematzadeh, Virginia Aglietti, Adam Fisch 외 arxiv

Moving beyond evaluations that collapse performance across heterogeneous prompts toward fine-grained evaluation at the prompt level, or within relatively homogeneous subsets, is necessary to diagnose generative models' s…