Classical Structured Prediction Losses for Sequence to Sequence Learning
There has been much recent work on training neural attention models at the sequence-level using either reinforcement learning-style methods or by optimizing the beam. In this paper, we survey a range of classical objective functions that have been widely used to train linear models for structured prediction and apply them to neural sequence to sequence models. Our experiments show that these losses can perform surprisingly well by slightly outperforming beam search optimization in a like for like setup. We also report new state of the art results on both IWSLT'14 German-English translation as well as Gigaword abstractive summarization. On the larger WMT'14 English-French translation task, sequence-level training achieves 41.5 BLEU which is on par with the state of the art.
Code (1)
Tasks
Abstractive Text SummarizationMachine TranslationPredictionReinforcement LearningReinforcement Learning (RL)Structured PredictionTranslationSimilar Papers 제목 키워드 기반
On Structured Prediction Theory with Calibrated Convex Surrogate Losses
We provide novel theoretical insights on structured prediction in the context of efficient convex surrogate loss minimization with consistency guarantees. For any task loss, we construct a convex surrogate that can be op…
PredictionStructured PredictionSecond Order Regret Bounds Against Generalized Expert Sequences under Partial Bandit Feedback
We study the problem of expert advice under partial bandit feedback setting and create a sequential minimax optimal algorithm. Our algorithm works with a more general partial monitoring setting, where, in contrast to the…
Compositional Generalization for Neural Semantic Parsing via Span-level Supervised Attention
We describe a span-level supervised attention loss that improves compositional generalization in semantic parsers. Our approach builds on existing losses that encourage attention maps in neural sequence-to-sequence model…
Machine TranslationSemantic ParsingTranslationWord AlignmentStructured Recommendation
Current recommender systems largely focus on static, unstructured content. In many scenarios, we would like to recommend content that has structure, such as a trajectory of points-of-interests in a city, or a playlist of…
Recommendation SystemsStructured PredictionvalidEfficient Gradient Computation for Structured Output Learning with Rational and Tropical Losses
Many structured prediction problems admit a natural loss function for evaluation such as the edit-distance or $n$-gram loss. However, existing learning algorithms are typically designed to optimize alternative objectives…
Structured Prediction