Offline Preference-Based Trajectory Evaluation

2026-06-16 · Fernando Diaz arxiv

Offline evaluation of agentic systems often collapses trajectories to terminal success, discarding information about partial progress and inducing widespread ties, creating substantial statistical inefficiency by reducing effective sample size and weakening the ability to distinguish systems. We propose preference-based trajectory evaluation, which compares trajectories directly through temporal preferences over progress and time-to-return profiles. We find that, across diverse agentic and interactive benchmarks, standard success-based metrics produce tied comparisons on roughly 75% of instances, whereas trajectory-aware preferences reduce ties to roughly 35%, improving discriminative power, ranking stability, and data efficiency. Our results suggest that benchmark saturation, often attributed to poor data collection or problem difficulty, may also be explained by the choice of evaluation measure.

📄 PDF Abstract BibTeX arXiv:2606.17541

Code (0)

등록된 구현이 없습니다.

In-Dataset Trajectory Return Regularization for Offline Preference-based Reinforcement Learning

2024-12-12 · Songjun Tu, Jingbo Sun, Qichao Zhang, Yaocheng Zhang 외

Offline preference-based reinforcement learning (PbRL) typically operates in two phases: first, use human preferences to learn a reward model and annotate rewards for a reward-free offline dataset; second, learn a policy…

Offline RL

Hindsight Preference Learning for Offline Preference-based Reinforcement Learning

2024-07-05 · Chen-Xiao Gao, Shengjun Fang, Chenjun Xiao, Yang Yu 외

Offline preference-based reinforcement learning (RL), which focuses on optimizing policies using human preferences between pairs of trajectory segments selected from an offline dataset, has emerged as a practical avenue …

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Provable Offline Preference-Based Reinforcement Learning

2023-05-24 · Wenhao Zhan, Masatoshi Uehara, Nathan Kallus, Jason D. Lee 외

In this paper, we investigate the problem of offline Preference-based Reinforcement Learning (PbRL) with human feedback where feedback is available in the form of preference between trajectory pairs rather than explicit …

reinforcement-learning Reinforcement Learning

When Does Trajectory-Level Supervision Permit Efficient Offline Reinforcement Learning?

2026-06-16 · Xuanfei Ren, Tengyang Xie arxiv

Offline reinforcement learning is typically analyzed under process-level reward supervision, yet many sequential decision datasets record only trajectory-level outcomes. We develop a statistical theory for offline policy…

Reinforcement Learning Offline RL

Policy-regularized Offline Multi-objective Reinforcement Learning

2024-01-04 · Qian Lin, Chao Yu, Zongkai Liu, Zifan Wu

In this paper, we aim to utilize only offline trajectory data to train a policy for multi-objective RL. We extend the offline policy-regularized method, a widely-adopted approach for single-objective offline RL problems,…

Multi-Objective Reinforcement Learning Offline RL reinforcement-learning Reinforcement Learning