How Does Distilled Data Complexity Impact the Quality and Confidence of Non-Autoregressive Machine Translation?
While non-autoregressive (NAR) models are showing great promise for machine translation, their use is limited by their dependence on knowledge distillation from autoregressive models. To address this issue, we seek to understand why distillation is so effective. Prior work suggests that distilled training data is less complex than manual translations. Based on experiments with the Levenshtein Transformer and the Mask-Predict NAR models on the WMT14 German-English task, this paper shows that different types of complexity have different impacts: while reducing lexical diversity and decreasing reordering complexity both help NAR learn better alignment between source and target, and thus improve translation quality, lexical diversity is the main reason why distillation increases model confidence, which affects the calibration of different NAR models differently.
Code (0)
등록된 구현이 없습니다.
Tasks
DiversityKnowledge DistillationMachine TranslationTranslationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Does Knowledge Distillation Matter for Large Language Model based Bundle Generation?
LLMs are increasingly explored for bundle generation, thanks to their reasoning capabilities and knowledge. However, deploying large-scale LLMs introduces significant efficiency challenges, primarily high computational c…
In-Context LearningKnowledge DistillationLanguage ModelingLanguage Modelling+1eDifFIQA: Towards Efficient Face Image Quality Assessment Based On Denoising Diffusion Probabilistic Models
State-of-the-art Face Recognition (FR) models perform well in constrained scenarios, but frequently fail in difficult real-world scenarios, when no quality guarantees can be made for face samples. For this reason, Face I…
DenoisingFace Image QualityFace Image Quality AssessmentFace Recognition+3Kandinsky 3.0 Technical Report
We present Kandinsky 3.0, a large-scale text-to-image generation model based on latent diffusion, continuing the series of text-to-image Kandinsky models and reflecting our progress to achieve higher quality and realism …
Image GenerationImage to Video GenerationSuper-ResolutionText to Image Generation+2An investigation of structures responsible for gender bias in BERT and DistilBERT
In recent years, large Transformer-based Pre-trained Language Models (PLM) have changed the Natural Language Processing (NLP) landscape, by pushing the performance boundaries of the state-of-the-art on a wide variety of …
AttributeFairnessAnalyzing Knowledge Distillation in Neural Machine Translation
Knowledge distillation has recently been successfully applied to neural machine translation. It allows for building shrunk networks while the resulting systems retain most of the quality of the original model. Despite th…
Knowledge DistillationMachine TranslationTranslation