Learning to Generalize from Sparse and Underspecified Rewards
We consider the problem of learning from sparse and underspecified rewards, where an agent receives a complex input, such as a natural language instruction, and needs to generate a complex response, such as an action sequence, while only receiving binary success-failure feedback. Such success-failure rewards are often underspecified: they do not distinguish between purposeful and accidental success. Generalization from underspecified rewards hinges on discounting spurious trajectories that attain accidental success, while learning from sparse feedback requires effective exploration. We address exploration by using a mode covering direction of KL divergence to collect a diverse set of successful trajectories, followed by a mode seeking KL divergence to train a robust policy. We propose Meta Reward Learning (MeRL) to construct an auxiliary reward function that provides more refined feedback for learning. The parameters of the auxiliary reward function are optimized with respect to the validation performance of a trained policy. The MeRL approach outperforms our alternative reward learning technique based on Bayesian Optimization, and achieves the state-of-the-art on weakly-supervised semantic parsing. It improves previous work by 1.2% and 2.4% on WikiTableQuestions and WikiSQL datasets respectively.
Code (1)
Tasks
Bayesian OptimizationSemantic ParsingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
ChatR1: Reinforcement Learning for Conversational Reasoning and Retrieval Augmented Question Answering
We present ChatR1, a reasoning framework based on reinforcement learning (RL) for conversational question answering (CQA). Reasoning plays an important role in CQA, where user intent evolves across dialogue turns, and ut…
Conversational Question AnsweringReinforcement LearningRobots That Know What to Ask: Recovering Misaligned Rewards through Targeted Explanations
Learning reward functions from demonstrations assumes that demonstrations provide adequate supervision over all features -- or task-relevant aspects of behavior. In practice, demonstrations are often imperfect: humans ma…
Learning Individual Intrinsic Reward in Multi-Agent Reinforcement Learning via Incorporating Generalized Human Expertise
Efficient exploration in multi-agent reinforcement learning (MARL) is a challenging problem when receiving only a team reward, especially in environments with sparse rewards. A powerful method to mitigate this issue invo…
Multi-agent Reinforcement LearningGeneralizing Across Multi-Objective Reward Functions in Deep Reinforcement Learning
Many reinforcement-learning researchers treat the reward function as a part of the environment, meaning that the agent can only know the reward of a state if it encounters that state in a trial run. However, we argue tha…
Deep Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)Effects of sparse rewards of different magnitudes in the speed of learning of model-based actor critic methods
Actor critic methods with sparse rewards in model-based deep reinforcement learning typically require a deterministic binary reward function that reflects only two possible outcomes: if, for each step, the goal has been …
Deep Reinforcement LearningMuJoCoReinforcement Learningvalid