Offline Reinforcement Learning for Large Scale Language Action Spaces
Training a task-oriented dialogue agent can be naturally formulated as offline reinforcement learning (RL) problem, where the agent aims to learn a conversational strategy to achieve user goals, only from a dialogue corpus. It is very challenging in terms of RL since the natural language action space is infinitely large, while feasible (syntactically and semantically correct) actions are very sparse. Thus, standard RL methods easily fail and generate responses diverging from human language, even when fine-tuning a powerful pre-trained language model. In this paper, we introduce GPT-Critic, an offline RL method for task-oriented dialogue. GPT-Critic is built upon GPT-2, fine-tuning the language model through behavior cloning of the critic-guided self-generated sentences. GPT-Critic is essentially free from the issue of diverging from human language since it learns from the sentences sampled from the pre-trained language model. In the experiments, we demonstrate that our algorithm outperforms the state-of-the-art in task-oriented dialogue benchmarks including MultiWOZ and ConvLab.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModelingLanguage ModellingOffline RLreinforcement-learningReinforcement LearningReinforcement Learning (RL)Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Contextual Transformer for Offline Meta Reinforcement Learning
The pretrain-finetuning paradigm in large-scale sequence models has made significant progress in natural language processing and computer vision tasks. However, such a paradigm is still hindered by several challenges in …
D4RLMeta Reinforcement LearningMuJoCoOffline RL+5Offline Reinforcement Learning With Combinatorial Action Spaces
Reinforcement learning problems often involve large action spaces arising from the simultaneous execution of multiple sub-actions, resulting in combinatorial action spaces. Learning in combinatorial action spaces is diff…
reinforcement-learningReinforcement LearningA Model-Based Reinforcement Learning with Adversarial Training for Online Recommendation
Reinforcement learning is effective in optimizing policies for recommender systems. Current solutions mostly focus on model-free approaches, which require frequent interactions with a real environment, and thus are expen…
Generative Adversarial NetworkModel-based Reinforcement LearningRecommendation Systemsreinforcement-learning+2Model-Based Reinforcement Learning with Adversarial Training for Online Recommendation
Reinforcement learning is well suited for optimizing policies of recommender systems. Current solutions mostly focus on model-free approaches, which require frequent interactions with the real environment, and thus are e…
Generative Adversarial NetworkModel-based Reinforcement LearningRecommendation Systemsreinforcement-learning+2QPO: Query-dependent Prompt Optimization via Multi-Loop Offline Reinforcement Learning
Prompt engineering has demonstrated remarkable success in enhancing the performance of large language models (LLMs) across diverse tasks. However, most existing prompt optimization methods only focus on the task-level pe…
BenchmarkingLanguage ModellingMathPrompt Engineering