Do all Roads Lead to Rome? Understanding the Role of Initialization in Iterative Back-Translation
Back-translation provides a simple yet effective approach to exploit monolingual corpora in Neural Machine Translation (NMT). Its iterative variant, where two opposite NMT models are jointly trained by alternately using a synthetic parallel corpus generated by the reverse model, plays a central role in unsupervised machine translation. In order to start producing sound translations and provide a meaningful training signal to each other, existing approaches rely on either a separate machine translation system to warm up the iterative procedure, or some form of pre-training to initialize the weights of the model. In this paper, we analyze the role that such initialization plays in iterative back-translation. Is the behavior of the final system heavily dependent on it? Or does iterative back-translation converge to a similar solution given any reasonable initialization? Through a series of empirical experiments over a diverse set of warmup systems, we show that, although the quality of the initial system does affect final performance, its effect is relatively small, as iterative back-translation has a strong tendency to convergence to a similar solution. As such, the margin of improvement left for the initialization method is narrow, suggesting that future research should focus more on improving the iterative mechanism itself.
Code (0)
등록된 구현이 없습니다.
Tasks
AllMachine TranslationNMTTranslationUnsupervised Machine TranslationSimilar Papers 제목 키워드 기반
Neuron Campaign for Initialization Guided by Information Bottleneck Theory
Initialization plays a critical role in the training of deep neural networks (DNN). Existing initialization strategies mainly focus on stabilizing the training process to mitigate gradient vanish/explosion problems. Howe…
Understanding the Learning Dynamics of LoRA: A Gradient Flow Perspective on Low-Rank Adaptation in Matrix Factorization
Despite the empirical success of Low-Rank Adaptation (LoRA) in fine-tuning pre-trained models, there is little theoretical understanding of how first-order methods with carefully crafted initialization adapt models to ne…
image-classificationImage ClassificationLaNet: Real-time Lane Identification by Learning Road SurfaceCharacteristics from Accelerometer Data
The resolution of GPS measurements, especially in urban areas, is insufficient for identifying a vehicle's lane. In this work, we develop a deep LSTM neural network model LaNet that determines the lane vehicles are on by…
ClassificationGeneral ClassificationCarSpeedNet: A Deep Neural Network-based Car Speed Estimation from Smartphone Accelerometer
We introduce the CarSpeedNet, a deep learning model designed to estimate car speed using three-axis accelerometer data from smartphones. Using 13 hours of data collected from a smartphone in cars across various roads, Ca…
All Roads Lead to Rome: Graph-Based Confidence Estimation for Large Language Model Reasoning
Confidence estimation is essential for the reliable deployment of large language models (LLMs). Existing methods are primarily designed for factual QA tasks and often fail to generalize to reasoning tasks. To address thi…