Active Preference-Based Gaussian Process Regression for Reward Learning

2020-05-06 · Erdem Biyik, Nicolas Huynh, Mykel J. Kochenderfer, Dorsa Sadigh

Designing reward functions is a challenging problem in AI and robotics. Humans usually have a difficult time directly specifying all the desirable behaviors that a robot needs to optimize. One common approach is to learn reward functions from collected expert demonstrations. However, learning reward functions from demonstrations introduces many challenges: some methods require highly structured models, e.g. reward functions that are linear in some predefined set of features, while others adopt less structured reward functions that on the other hand require tremendous amount of data. In addition, humans tend to have a difficult time providing demonstrations on robots with high degrees of freedom, or even quantifying reward values for given demonstrations. To address these challenges, we present a preference-based learning approach, where as an alternative, the human feedback is only in the form of comparisons between trajectories. Furthermore, we do not assume highly constrained structures on the reward function. Instead, we model the reward function using a Gaussian Process (GP) and propose a mathematical formulation to actively find a GP using only human preferences. Our approach enables us to tackle both inflexibility and data-inefficiency problems within a preference-based learning framework. Our results in simulations and a user study suggest that our approach can efficiently learn expressive reward functions for robotics tasks.

📄 PDF Abstract BibTeX arXiv:2005.02575

Code (1)

Stanford-ILIAD/active-preference-based-gpr 공식 구현

Tasks

regression

Methods 이 논문이 사용한 방법론

Gaussian Process Gaussian Processes are non-parametric models for approximating functions. They rely upon a measure of similarity between points (the kernel function) to predict the value for…

A unified framework for closed-form nonparametric regression, classification, preference and mixed problems with Skew Gaussian Processes

2020-12-12 · Alessio Benavoli, Dario Azzimonti, Dario Piga

Skew-Gaussian processes (SkewGPs) extend the multivariate Unified Skew-Normal distributions over finite dimensional vectors to distribution over functions. SkewGPs are more general and flexible than Gaussian processes, a…

Active Learning Binary Classification Form Gaussian Processes+1

Bayesian Active Learning for Classification and Preference Learning

2011-12-24 · Neil Houlsby, Ferenc Huszár, Zoubin Ghahramani, Máté Lengyel

Information theoretic active learning has been widely studied for probabilistic models. For simple regression an optimal myopic policy is easily tractable. However, for other tasks and with more complex models, such as c…

Active Learning Classification General Classification

Drifting Preference Optimization for One-Step Generative Models

2026-06-01 · Zhou Jiang, Yandong Wen, Zhen Liu arxiv

One-step text-to-image generators are attractive for deployment because they generate an image with a single forward pass, but preference finetuning them remains difficult: standard alignment methods often rely on policy…

Batch Active Learning of Reward Functions from Human Preferences

2024-02-24 · Erdem Biyik, Nima Anari, Dorsa Sadigh

Data generation and labeling are often expensive in robot learning. Preference-based learning is a concept that enables reliable labeling by querying users with preference questions. Active querying methods are commonly …

Active Learning Point Processes

Neuronal Gaussian Process Regression

2020-12-01 · NeurIPS 2020 12 · Johannes Friedrich

The brain takes uncertainty intrinsic to our world into account. For example, associating spatial locations with rewards requires to predict not only expected reward at new spatial locations but also its uncertainty to a…