Differentiable Scheduled Sampling for Credit Assignment
We demonstrate that a continuous relaxation of the argmax operation can be used to create a differentiable approximation to greedy decoding for sequence-to-sequence (seq2seq) models. By incorporating this approximation into the scheduled sampling training procedure (Bengio et al., 2015)--a well-known technique for correcting exposure bias--we introduce a new training objective that is continuous and differentiable everywhere and that can provide informative gradients near points where previous decoding decisions change their value. In addition, by using a related approximation, we demonstrate a similar approach to sampled-based training. Finally, we show that our approach outperforms cross-entropy training and scheduled sampling procedures in two sequence prediction tasks: named entity recognition and machine translation.
Code (0)
등록된 구현이 없습니다.
Tasks
Machine Translationnamed-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)TranslationSimilar Papers 제목 키워드 기반
Differentiable Sampling with Flexible Reference Word Order for Neural Machine Translation
Despite some empirical success at correcting exposure bias in machine translation, scheduled sampling algorithms suffer from a major drawback: they incorrectly assume that words in the reference translations and in sampl…
Machine TranslationTranslationA Review of Neuroscience-Inspired Machine Learning
One major criticism of deep learning centers around the biological implausibility of the credit assignment schema used for learning -- backpropagation of errors. This implausibility translates into practical limitations,…
Credit Assignment with Resets in Language Model Reasoning
Contemporary reinforcement learning with verifiable reward methods post-train language models on multi-step reasoning by assigning a single outcome reward uniformly across all tokens in a trajectory. Such uniform assignm…
Reinforcement LearningDueling Posterior Sampling for Preference-Based Reinforcement Learning
In preference-based reinforcement learning (RL), an agent interacts with the environment while receiving preferences instead of absolute feedback. While there is increasing research activity in preference-based RL, the d…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Shapley Counterfactual Credits for Multi-Agent Reinforcement Learning
Centralized Training with Decentralized Execution (CTDE) has been a popular paradigm in cooperative Multi-Agent Reinforcement Learning (MARL) settings and is widely used in many real applications. One of the major challe…
counterfactualMulti-agent Reinforcement Learningreinforcement-learningReinforcement Learning+3