An Actor-Critic Algorithm for Sequence Prediction
We present an approach to training neural networks to generate sequences using actor-critic methods from reinforcement learning (RL). Current log-likelihood training methods are limited by the discrepancy between their training and testing modes, as models must generate tokens conditioned on their previous guesses rather than the ground-truth tokens. We address this problem by introducing a \textit{critic} network that is trained to predict the value of an output token, given the policy of an \textit{actor} network. This results in a training procedure that is much closer to the test phase, and allows us to directly optimize for a task-specific score such as BLEU. Crucially, since we leverage these techniques in the supervised learning setting rather than the traditional RL setting, we condition the critic network on the ground-truth output. We show that our method leads to improved performance on both a synthetic task, and for German-English machine translation. Our analysis paves the way for such methods to be applied in natural language generation tasks, such as machine translation, caption generation, and dialogue modelling.
Code (3)
Tasks
Caption GenerationMachine TranslationPredictionReinforcement LearningReinforcement Learning (RL)Spelling CorrectionText GenerationTranslationSimilar Papers 제목 키워드 기반
From Credit Assignment to Entropy Regularization: Two New Algorithms for Neural Sequence Prediction
In this work, we study the credit assignment problem in reward augmented maximum likelihood (RAML) learning, and establish a theoretical equivalence between the token-level counterpart of RAML and the entropy regularized…
Predictionreinforcement-learningReinforcement LearningReinforcement Learning (RL)Efficient Sequence Labeling with Actor-Critic Training
Neural approaches to sequence labeling often use a Conditional Random Field (CRF) to model their output dependencies, while Recurrent Neural Networks (RNN) are used for the same purpose in other tasks. We set out to esta…
Decision MakingNERSequential Decision MakingTransliterationAFU: Actor-Free critic Updates in off-policy RL for continuous control
This paper presents AFU, an off-policy deep RL algorithm addressing in a new way the challenging "max-Q problem" in Q-learning for continuous action spaces, with a solution based on regression and conditional gradient sc…
continuous-controlContinuous ControlQ-LearningDecision-Aware Actor-Critic with Function Approximation and Theoretical Guarantees
Actor-critic (AC) methods are widely used in reinforcement learning (RL) and benefit from the flexibility of using any policy gradient method as the actor and value-based method as the critic. The critic is usually train…
Reinforcement Learning (RL)Investigating Critical Risk Factors in Liver Cancer Prediction
We exploit liver cancer prediction model using machine learning algorithms based on epidemiological data of over 55 thousand peoples from 2014 to the present. The best performance is an AUC of 0.71. We analyzed model par…
BIG-bench Machine LearningPrediction