Human-in-the-Loop Policy Optimization for Preference-Based Multi-Objective Reinforcement Learning
Multi-objective reinforcement learning (MORL) aims to find a set of high-performing and diverse policies that address trade-offs between multiple conflicting objectives. However, in practice, decision makers (DMs) often deploy only one or a limited number of trade-off policies. Providing too many diversified trade-off policies to the DM not only significantly increases their workload but also introduces noise in multi-criterion decision-making. With this in mind, we propose a human-in-the-loop policy optimization framework for preference-based MORL that interactively identifies policies of interest. Our method proactively learns the DM's implicit preference information without requiring any a priori knowledge, which is often unavailable in real-world black-box decision scenarios. The learned preference information is used to progressively guide policy optimization towards policies of interest. We evaluate our approach against three conventional MORL algorithms that do not consider preference information and four state-of-the-art preference-based MORL algorithms on two MORL environments for robot control and smart grid management. Experimental results fully demonstrate the effectiveness of our proposed method in comparison to the other peer algorithms.
Code (0)
등록된 구현이 없습니다.
Tasks
Decision MakingManagementMulti-Objective Reinforcement LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
TakeAD: Preference-based Post-optimization for End-to-end Autonomous Driving with Expert Takeover Data
Existing end-to-end autonomous driving methods typically rely on imitation learning (IL) but face a key challenge: the misalignment between open-loop training and closed-loop deployment. This misalignment often triggers …
Autonomous DrivingIs On-Policy Data always the Best Choice for Direct Preference Optimization-based LM Alignment?
The alignment of language models~(LMs) with human preferences is critical for building reliable AI systems. The problem is typically framed as optimizing an LM policy to maximize the expected reward that reflects human p…
Deployable Vision-driven UAV River Navigation via Human-in-the-loop Preference Alignment
Rivers are critical corridors for environmental monitoring and disaster response, where Unmanned Aerial Vehicles (UAVs) guided by vision-driven policies can provide fast, low-cost coverage. However, deployment exposes si…
Reinforcement LearningDrive As You Like: Multi-Head Diffusion with Reinforcement Learning for Personalized Driving
Despite significant progress, imitation learning-based autonomous driving planners remain largely restricted to reproducing high-frequency biased behaviors, overlooking the inherent behavioral diversity of human driving.…
Autonomous DrivingMotion PlanningGEM: Generative Entropy-Guided Preference Modeling for Few-shot Alignment of LLMs
Alignment of large language models (LLMs) with human preferences typically relies on supervised reward models or external judges that demand abundant annotations. However, in fields that rely on professional knowledge, s…
Mathematical ReasoningDecision Making