Does the Order of Training Samples Matter? Improving Neural Data-to-Text Generation with Curriculum Learning
Recent advancements in data-to-text generation largely take on the form of neural end-to-end systems. Efforts have been dedicated to improving text generation systems by changing the order of training samples in a process known as curriculum learning. Past research on sequence-to-sequence learning showed that curriculum learning helps to improve both the performance and convergence speed. In this work, we delve into the same idea surrounding the training samples consisting of structured data and text pairs, where at each update, the curriculum framework selects training samples based on the model's competence. Specifically, we experiment with various difficulty metrics and put forward a soft edit distance metric for ranking training samples. Our benchmarks show faster convergence speed where training time is reduced by 38.7% and performance is boosted by 4.84 BLEU.
Code (0)
등록된 구현이 없습니다.
Tasks
Data-to-Text GenerationText GenerationSimilar Papers 제목 키워드 기반
Impact of Strategic Sampling and Supervision Policies on Semi-supervised Learning
In semi-supervised representation learning frameworks, when the number of labelled data is very scarce, the quality and representativeness of these samples become increasingly important. Existing literature on semi-super…
Representation LearningA deep-learning algorithm to disentangle self-interacting dark matter and AGN feedback models
Different models of dark matter can alter the distribution of mass in galaxy clusters in a variety of ways. However, so can uncertain astrophysical feedback mechanisms. Here we present a Machine Learning method that ''le…
Tailoring Instructions to Student's Learning Levels Boosts Knowledge Distillation
It has been commonly observed that a teacher model with superior performance does not necessarily result in a stronger student, highlighting a discrepancy between current teacher training practices and effective knowledg…
Knowledge Distillationtext-classificationText ClassificationTransfer LearningWhen classifying arguments, BERT doesn’t care about word order...except when it matters
Does Order Matter? An Empirical Study on Generating Multiple Keyphrases as a Sequence
Recently, concatenating multiple keyphrases as a target sequence has been proposed as a new learning paradigm for keyphrase generation. Existing studies concatenate target keyphrases in different orders but no study has …
Keyphrase Generation