BARThez: a Skilled Pretrained French Sequence-to-Sequence Model
Inductive transfer learning has taken the entire NLP field by storm, with models such as BERT and BART setting new state of the art on countless NLU tasks. However, most of the available models and research have been conducted for English. In this work, we introduce BARThez, the first large-scale pretrained seq2seq model for French. Being based on BART, BARThez is particularly well-suited for generative tasks. We evaluate BARThez on five discriminative tasks from the FLUE benchmark and two generative tasks from a novel summarization dataset, OrangeSum, that we created for this research. We show BARThez to be very competitive with state-of-the-art BERT-based French language models such as CamemBERT and FlauBERT. We also continue the pretraining of a multilingual BART on BARThez' corpus, and show our resulting model, mBARThez, to significantly boost BARThez' generative performance. Code, data and models are publicly available.
Code (5)
Tasks
FLUENatural Language UnderstandingOrangeSumSelf-Supervised LearningText SummarizationTransfer LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Effectiveness of French Language Models on Abstractive Dialogue Summarization Task
Pre-trained language models have established the state-of-the-art on various natural language processing tasks, including dialogue summarization, which allows the reader to quickly access key information from long conver…
Abstractive Dialogue SummarizationAdaptation of Biomedical and Clinical Pretrained Models to French Long Documents: A Comparative Study
Recently, pretrained language models based on BERT have been introduced for the French biomedical domain. Although these models have achieved state-of-the-art results on biomedical and clinical NLP tasks, they are constr…
named-entity-recognitionNamed Entity RecognitionUsing Multimodal and Language-Agnostic Sentence Embeddings for Abstractive Summarization
Abstractive summarization aims to generate concise summaries by creating new sentences, allowing for flexible rephrasing. However, this approach can be vulnerable to inaccuracies, particularly `hallucinations' where the …
GAFSV-Net: A Vision Framework for Online Signature Verification
Online signature verification (OSV) requires distinguishing skilled forgeries from genuine samples under high intra-class variability and with very few enrollment samples. Existing deep learning methods operate directly …
Temporal SequencesLow-resource Neural Machine Translation: Benchmarking State-of-the-art Transformer for Wolof<->French
In this paper, we propose two neural machine translation (NMT) systems (French-to-Wolof and Wolof-to-French) based on sequence-to-sequence with attention and Transformer architectures. We trained our models on the parall…
BenchmarkingLow Resource Neural Machine TranslationLow-Resource Neural Machine TranslationMachine Translation+3