A State Augmentation based approach to Reinforcement Learning from Human Preferences

2023-02-17 · Mudit Verma, Subbarao Kambhampati

Reinforcement Learning has suffered from poor reward specification, and issues for reward hacking even in simple enough domains. Preference Based Reinforcement Learning attempts to solve the issue by utilizing binary feedbacks on queried trajectory pairs by a human in the loop indicating their preferences about the agent's behavior to learn a reward model. In this work, we present a state augmentation technique that allows the agent's reward model to be robust and follow an invariance consistency that significantly improved performance, i.e. the reward recovery and subsequent return computed using the learned policy over our baseline PEBBLE. We validate our method on three domains, Mountain Car, a locomotion task of Quadruped-Walk, and a robotic manipulation task of Sweep-Into, and find that using the proposed augmentation the agent not only benefits in the overall performance but does so, quite early in the agent's training phase.

📄 PDF Abstract BibTeX arXiv:2302.08734

Code (0)

등록된 구현이 없습니다.

Tasks

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

It Takes Two: On the Seamlessness between Reward and Policy Model in RLHF

2024-06-12 · Taiming Lu, Lingfeng Shen, Xinyu Yang, Weiting Tan 외

Reinforcement Learning from Human Feedback (RLHF) involves training policy models (PMs) and reward models (RMs) to align language models with human preferences. Instead of focusing solely on PMs and RMs independently, we…

Hindsight Preference Replay Improves Preference-Conditioned Multi-Objective Reinforcement Learning

2026-01-08 · Jonaid Shianifar, Michael Schukat, Karl Mason arxiv

Multi-objective reinforcement learning (MORL) enables agents to optimize vector-valued rewards while respecting user preferences. CAPQL, a preference-conditioned actor-critic method, achieves this by conditioning on weig…

Reinforcement Learning

Warmth and competence in human-agent cooperation

2022-01-31 · Kevin R. McKee, Xuechunzi Bai, Susan T. Fiske

Interaction and cooperation with humans are overarching aspirations of artificial intelligence (AI) research. Recent studies demonstrate that AI agents trained with deep reinforcement learning are capable of collaboratin…

Deep Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

MotionRL: Align Text-to-Motion Generation to Human Preferences with Multi-Reward Reinforcement Learning

2024-10-09 · Xiaoyang Liu, Yunyao Mao, Wengang Zhou, Houqiang Li

We introduce MotionRL, the first approach to utilize Multi-Reward Reinforcement Learning (RL) for optimizing text-to-motion generation tasks and aligning them with human preferences. Previous works focused on improving n…

Motion Generation reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

An offline approach to fNIRS-guided reinforcement learning for robot behavior

2026-07-15 · Julia Santaniello, Madelaine Brower, Benson Jiang, Donatello Sassaroli 외 arxiv

Human-in-the-loop Reinforcement Learning has become a popular approach for training, finetuning, and aligning robot behavior with user preferences. Our paper explores the feasibility of using brain signals via functional…