Transfer Reward Learning for Policy Gradient-Based Text Generation
Task-specific scores are often used to optimize for and evaluate the performance of conditional text generation systems. However, such scores are non-differentiable and cannot be used in the standard supervised learning paradigm. Hence, policy gradient methods are used since the gradient can be computed without requiring a differentiable objective. However, we argue that current n-gram overlap based measures that are used as rewards can be improved by using model-based rewards transferred from tasks that directly compare the similarity of sentence pairs. These reward models either output a score of sentence-level syntactic and semantic similarity between entire predicted and target sentences as the expected return, or for intermediate phrases as segmented accumulative rewards. We demonstrate that using a \textit{Transferable Reward Learner} leads to improved results on semantical evaluation measures in policy-gradient models for image captioning tasks. Our InferSent actor-critic model improves over a BLEU trained actor-critic model on MSCOCO when evaluated on a Word Mover's Distance similarity measure by 6.97 points, also improving on a Sliding Window Cosine Similarity measure by 10.48 points. Similar performance improvements are also obtained on the smaller Flickr-30k dataset, demonstrating the general applicability of the proposed transfer learning method.
Code (0)
등록된 구현이 없습니다.
Tasks
Conditional Text GenerationImage CaptioningPolicy Gradient MethodsSemantic SimilaritySemantic Textual SimilaritySentenceText GenerationTransfer LearningSimilar Papers 제목 키워드 기반
Toward Diverse Text Generation with Inverse Reinforcement Learning
Text generation is a crucial task in NLP. Recently, several adversarial generative models have been proposed to improve the exposure bias problem in text generation. Though these models gain great success, they still suf…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Text GenerationContrastive Policy Gradient: Aligning LLMs on sequence-level scores in a supervised-friendly fashion
Reinforcement Learning (RL) has been used to finetune Large Language Models (LLMs) using a reward model trained from preference data, to better align with human judgment. The recently introduced direct alignment methods,…
Code GenerationNatural Language InferenceReinforcement Learning (RL)ARAML: A Stable Adversarial Training Framework for Text Generation
Most of the existing generative adversarial networks (GAN) for text generation suffer from the instability of reinforcement learning training algorithms such as policy gradient, leading to unstable performance. To tackle…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Text GenerationLearning from Bootstrapping and Stepwise Reinforcement Reward: A Semi-Supervised Framework for Text Style Transfer
Text style transfer is an important task in controllable language generation. Supervised approaches have pushed performance improvement on style-oriented rewriting such as formality conversion. However, challenges remain…
SentenceStyle TransferText GenerationText Style TransferText Generation with Efficient (Soft) $Q$-Learning
Maximum likelihood estimation (MLE) is the predominant algorithm for training text generation models. This paradigm relies on direct supervision examples, which is not applicable to many emerging applications, such as ge…
Q-LearningReinforcement Learning (RL)Text Generation