Imperfect also Deserves Reward: Multi-Level and Sequential Reward Modeling for Better Dialog Management
For task-oriented dialog systems, training a Reinforcement Learning (RL) based Dialog Management module suffers from low sample efficiency and slow convergence speed due to the sparse rewards in RL.To solve this problem, many strategies have been proposed to give proper rewards when training RL, but their rewards lack interpretability and cannot accurately estimate the distribution of state-action pairs in real dialogs. In this paper, we propose a multi-level reward modeling approach that factorizes a reward into a three-level hierarchy: domain, act, and slot. Based on inverse adversarial reinforcement learning, our designed reward model can provide more accurate and explainable reward signals for state-action pairs.Extensive evaluations show that our approach can be applied to a wide range of reinforcement learning-based dialog systems and significantly improves both the performance and the speed of convergence.
Code (1)
Tasks
Managementreinforcement-learningReinforcement LearningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Mind the Gap: Offline Policy Optimization for Imperfect Rewards
Reward function is essential in reinforcement learning (RL), serving as the guiding signal to incentivize agents to solve given tasks, however, is also notoriously difficult to design. In many cases, only imperfect rewar…
Reinforcement Learning (RL)Hierarchical Apprenticeship Learning from Imperfect Demonstrations with Evolving Rewards
While apprenticeship learning has shown promise for inducing effective pedagogical policies directly from student interactions in e-learning environments, most existing approaches rely on optimal or near-optimal expert d…
RATE: Causal Explainability of Reward Models with Imperfect Counterfactuals
Reward models are widely used as proxies for human preferences when aligning or evaluating LLMs. However, reward models are black boxes, and it is often unclear what, exactly, they are actually rewarding. In this paper w…
AttributeLanguage ModelingLanguage ModellingSensitivityOn Lai's Upper Confidence Bound in Multi-Armed Bandits
In this memorial paper, we honor Tze Leung Lai's seminal contributions to the topic of multi-armed bandits, with a specific focus on his pioneering work on the upper confidence bound. We establish sharp non-asymptotic re…
Multi-Armed BanditsReinforcement Learning From Imperfect Corrective Actions And Proxy Rewards
In practice, reinforcement learning (RL) agents are often trained with a possibly imperfect proxy reward function, which may lead to a human-agent alignment issue (i.e., the learned policy either converges to non-optimal…
Atari GamesAutonomous Drivingreinforcement-learningReinforcement Learning+1