Fast Sequence Generation with Multi-Agent Reinforcement Learning
Autoregressive sequence Generation models have achieved state-of-the-art performance in areas like machine translation and image captioning. These models are autoregressive in that they generate each word by conditioning on previously generated words, which leads to heavy latency during inference. Recently, non-autoregressive decoding has been proposed in machine translation to speed up the inference time by generating all words in parallel. Typically, these models use the word-level cross-entropy loss to optimize each word independently. However, such a learning process fails to consider the sentence-level consistency, thus resulting in inferior generation quality of these non-autoregressive models. In this paper, we propose a simple and efficient model for Non-Autoregressive sequence Generation (NAG) with a novel training paradigm: Counterfactuals-critical Multi-Agent Learning (CMAL). CMAL formulates NAG as a multi-agent reinforcement learning system where element positions in the target sequence are viewed as agents that learn to cooperatively maximize a sentence-level reward. On MSCOCO image captioning benchmark, our NAG method achieves a performance comparable to state-of-the-art autoregressive models, while brings 13.9x decoding speedup. On WMT14 EN-DE machine translation dataset, our method outperforms cross-entropy trained baseline by 6.0 BLEU points while achieves the greatest decoding speedup of 17.46x.
Code (0)
등록된 구현이 없습니다.
Tasks
Image CaptioningMachine TranslationMulti-agent Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)SentenceTranslationSimilar Papers 제목 키워드 기반
Hybrid Reinforcement Learning with Expert State Sequences
Existing imitation learning approaches often require that the complete demonstration data, including sequences of actions and states, are available. In this paper, we consider a more realistic and difficult scenario wher…
Atari GamesImitation Learningreinforcement-learningReinforcement Learning+1Multi-Agent Reinforcement Learning Meets Leaf Sequencing in Radiotherapy
In contemporary radiotherapy planning (RTP), a key module leaf sequencing is predominantly addressed by optimization-based approaches. In this paper, we propose a novel deep reinforcement learning (DRL) model termed as R…
Deep Reinforcement LearningMulti-agent Reinforcement Learningreinforcement-learningReinforcement LearningMulti-agent cooperation through in-context co-player inference
Achieving cooperation among self-interested agents remains a fundamental challenge in multi-agent reinforcement learning. Recent work showed that mutual cooperation can be induced between "learning-aware" agents that acc…
Multi-agent Reinforcement LearningImproving Token-Based World Models with Parallel Observation Prediction
Motivated by the success of Transformers when applied to sequences of discrete symbols, token-based world models (TBWMs) were recently proposed as sample-efficient methods. In TBWMs, the world model consumes agent experi…
GPUPredictionStructured State Space Models for In-Context Reinforcement Learning
Structured state space sequence (S4) models have recently achieved state-of-the-art performance on long-range sequence modeling tasks. These models also have fast inference speeds and parallelisable training, making them…
continuous-controlContinuous ControlIn-Context Reinforcement LearningMeta-Learning+3