Big Bidirectional Insertion Representations for Documents
The Insertion Transformer is well suited for long form text generation due to its parallel generation capabilities, requiring $O(\log_2 n)$ generation steps to generate $n$ tokens. However, modeling long sequences is difficult, as there is more ambiguity captured in the attention mechanism. This work proposes the Big Bidirectional Insertion Representations for Documents (Big BIRD), an insertion-based model for document-level translation tasks. We scale up the insertion-based models to long form documents. Our key contribution is introducing sentence alignment via sentence-positional embeddings between the source and target document. We show an improvement of +4.3 BLEU on the WMT'19 English$\rightarrow$German document-level translation task compared with the Insertion Transformer baseline.
Code (0)
등록된 구현이 없습니다.
Tasks
FormSentenceText GenerationTranslationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
LayoutBERT: Masked Language Layout Model for Object Insertion
Image compositing is one of the most fundamental steps in creative workflows. It involves taking objects/parts of several images to create a new image, called a composite. Currently, this process is done manually by crea…
Language ModellingmodelObjectRetrievalDuration-aware pause insertion using pre-trained language model for multi-speaker text-to-speech
Pause insertion, also known as phrase break prediction and phrasing, is an essential part of TTS systems because proper pauses with natural duration significantly enhance the rhythm and intelligibility of synthetic speec…
Language ModelingLanguage ModellingPredictionRhythm+2Multi-level analysis and recognition of the text sentiment on the example of consumer opinions
In this article, we present a novel multi-domain dataset of Polish text reviews, annotated with sentiment on different levels: sentences and the whole documents. The annotation was made by linguists in a 2+1 scheme (with…
General ClassificationregressionIntelligent bidirectional rapidly-exploring random trees for optimal motion planning in complex cluttered environments
The sampling based motion planning algorithm known as Rapidly-exploring Random Trees (RRT) has gained the attention of many researchers due to their computational efficiency and effectiveness. Recently, a variant of RRT …
Computational EfficiencyMotion PlanningOptimal Motion PlanningA Unified Tagging Solution: Bidirectional LSTM Recurrent Neural Network with Word Embedding
Bidirectional Long Short-Term Memory Recurrent Neural Network (BLSTM-RNN) has been shown to be very effective for modeling and predicting sequential data, e.g. speech utterances or handwritten documents. In this study, w…
ChunkingFeature Engineeringnamed-entity-recognitionNamed Entity Recognition+2