Efficient Meta Reinforcement Learning for Preference-based Fast Adaptation
Learning new task-specific skills from a few trials is a fundamental challenge for artificial intelligence. Meta reinforcement learning (meta-RL) tackles this problem by learning transferable policies that support few-shot adaptation to unseen tasks. Despite recent advances in meta-RL, most existing methods require the access to the environmental reward function of new tasks to infer the task objective, which is not realistic in many practical applications. To bridge this gap, we study the problem of few-shot adaptation in the context of human-in-the-loop reinforcement learning. We develop a meta-RL algorithm that enables fast policy adaptation with preference-based feedback. The agent can adapt to new tasks by querying human's preference between behavior trajectories instead of using per-step numeric rewards. By extending techniques from information theory, our approach can design query sequences to maximize the information gain from human interactions while tolerating the inherent error of non-expert human oracle. In experiments, we extensively evaluate our method, Adaptation with Noisy OracLE (ANOLE), on a variety of meta-RL benchmark tasks and demonstrate substantial improvement over baseline algorithms in terms of both feedback efficiency and error tolerance.
Code (1)
Tasks
Meta Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Meta Policy Learning for Cold-Start Conversational Recommendation
Conversational recommender systems (CRS) explicitly solicit users' preferences for improved recommendations on the fly. Most existing CRS solutions count on a single policy trained by reinforcement learning for a populat…
Conversational RecommendationMeta Reinforcement LearningRecommendation Systemsreinforcement-learning+2Offline Meta-level Model-based Reinforcement Learning Approach for Cold-Start Recommendation
Reinforcement learning (RL) has shown great promise in optimizing long-term user interest in recommender systems. However, existing RL-based recommendation methods need a large number of interactions for each user to lea…
Model-based Reinforcement LearningRecommendation Systemsreinforcement-learningReinforcement Learning+1One Adapts to Any: Meta Reward Modeling for Personalized LLM Alignment
Alignment of Large Language Models (LLMs) aims to align outputs with human preferences, and personalized alignment further adapts models to individual users. This relies on personalized reward models that capture user-sp…
Hypothesis Network Planned Exploration for Rapid Meta-Reinforcement Learning Adaptation
Meta Reinforcement Learning (Meta RL) trains agents that adapt to fast-changing environments and tasks. Current strategies often lose adaption efficiency due to the passive nature of model exploration, causing delayed un…
Meta Reinforcement Learningreinforcement-learningReinforcement LearningBayesian Meta-reinforcement Learning for Traffic Signal Control
In recent years, there has been increasing amount of interest around meta reinforcement learning methods for traffic signal control, which have achieved better performance compared with traditional control methods. Howev…
Continual LearningMeta-LearningMeta Reinforcement Learningreinforcement-learning+3