Surprisingly Easy Hard-Attention for Sequence to Sequence Learning
In this paper we show that a simple beam approximation of the joint distribution between attention and output is an easy, accurate, and efficient attention mechanism for sequence to sequence learning. The method combines the advantage of sharp focus in hard attention and the implementation ease of soft attention. On five translation tasks we show effortless and consistent gains in BLEU compared to existing attention mechanisms.
Code (1)
Tasks
Hard AttentionImage CaptioningMorphological InflectionTranslationSimilar Papers 제목 키워드 기반
Hungry Hungry Hippos: Towards Language Modeling with State Space Models
State space models (SSMs) have demonstrated state-of-the-art sequence modeling performance in some modalities, but underperform attention in language modeling. Moreover, despite scaling nearly linearly in sequence length…
8kCoreference ResolutionFew-Shot LearningLanguage Modeling+6Learning Online Alignments with Continuous Rewards Policy Gradient
Sequence-to-sequence models with soft attention had significant success in machine translation, speech recognition, and question answering. Though capable and easy to use, they require that the entirety of the input sequ…
Machine TranslationQuestion Answeringspeech-recognitionSpeech Recognition+2Transformers as Transducers
We study the sequence-to-sequence mapping capacity of transformers by relating them to finite transducers, and find that they can express surprisingly large classes of transductions. We do so using variants of RASP, a pr…
Hard AttentionPOSLearning What's Easy: Fully Differentiable Neural Easy-First Taggers
We introduce a novel neural easy-first decoder that learns to solve sequence tagging tasks in a flexible order. In contrast to previous easy-first decoders, our models are end-to-end differentiable. The decoder iterative…
DecoderImitation LearningNamed Entity Recognition (NER)Part-Of-Speech TaggingThe Unreasonable Effectiveness of Easy Training Data for Hard Tasks
How can we train models to perform well on hard test data when hard training data is by definition difficult to label correctly? This question has been termed the scalable oversight problem and has drawn increasing atten…
General KnowledgeIn-Context LearningQuestion Answering