FeatureBART: Feature Based Sequence-to-Sequence Pre-Training for Low-Resource NMT
In this paper we present FeatureBART, a linguistically motivated sequence-to-sequence monolingual pre-training strategy in which syntactic features such as lemma, part-of-speech and dependency labels are incorporated into the span prediction based pre-training framework (BART). These automatically extracted features are incorporated via approaches such as concatenation and relevance mechanisms, among which the latter is known to be better than the former. When used for low-resource NMT as a downstream task, we show that these feature based models give large improvements in bilingual settings and modest ones in multilingual settings over their counterparts that do not use features.
Code (0)
등록된 구현이 없습니다.
Tasks
LEMMALow Resource NMTNMTSimilar Papers 제목 키워드 기반
Analysis of Multilingual Sequence-to-Sequence speech recognition systems
This paper investigates the applications of various multilingual approaches developed in conventional hidden Markov model (HMM) systems to sequence-to-sequence (seq2seq) automatic speech recognition (ASR). On a set compo…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Sequence-To-Sequence Speech Recognitionspeech-recognition+2DeepExpress: Heterogeneous and Coupled Sequence Modeling for Express Delivery Prediction
The prediction of express delivery sequence, i.e., modeling and estimating the volumes of daily incoming and outgoing parcels for delivery, is critical for online business, logistics, and positive customer experience, an…
PredictionLow Resource Multi-Task Sequence Tagging -- Revisiting Dynamic Conditional Random Fields
We compare different models for low resource multi-task sequence tagging that leverage dependencies between label sequences for different tasks. Our analysis is aimed at datasets where each example has labels for multipl…
Multi-Task LearningSentenceDynamic Adaptive Shared Experts with Grouped Multi-Head Attention Mixture of Experts
Transformer models based on the Mixture of Experts (MoE) architecture have made significant progress in long-sequence modeling, but existing models still have shortcomings in computational efficiency and the ability to c…
Computational EfficiencySequence to sequence pretraining for a less-resourced Slovenian language
Large pretrained language models have recently conquered the area of natural language processing. As an alternative to predominant masked language modelling introduced in BERT, the T5 model has introduced a more general …
Language ModelingLanguage ModellingMachine TranslationOpen-Domain Question Answering+4