ReWE: Regressing Word Embeddings for Regularization of Neural Machine Translation Systems
Regularization of neural machine translation is still a significant problem, especially in low-resource settings. To mollify this problem, we propose regressing word embeddings (ReWE) as a new regularization technique in a system that is jointly trained to predict the next word in the translation (categorical value) and its word embedding (continuous value). Such a joint training allows the proposed system to learn the distributional properties represented by the word embeddings, empirically improving the generalization to unseen sentences. Experiments over three translation datasets have showed a consistent improvement over a strong baseline, ranging between 0.91 and 2.54 BLEU points, and also a marked improvement over a state-of-the-art system.
Code (0)
등록된 구현이 없습니다.
Tasks
Machine TranslationTranslationWord EmbeddingsSimilar Papers 제목 키워드 기반
Regressing Word and Sentence Embeddings for Regularization of Neural Machine Translation
In recent years, neural machine translation (NMT) has become the dominant approach in automated translation. However, like many other deep learning approaches, NMT suffers from overfitting when the amount of training dat…
ClusteringMachine TranslationNMTSentence+2BULNER: BUg Localization with word embeddings and NEtwork Regularization
Bug localization (BL) from the bug report is the strategic activity of the software maintaining process. Because BL is a costly and tedious activity, BL techniques information retrieval-based and machine learning-based c…
BIG-bench Machine LearningInformation RetrievalRetrievalWord EmbeddingsContrastive Word Embedding Learning for Neural Machine Translation
Seq2seq models have shined in the field of Neural Machine Translation (NMT). However, word embeddings learned by NMT models tend to degenerate and be distributed into a narrow cone, named {\em{representation degeneration…
Machine TranslationNMTRepresentation LearningSemantic Similarity+3Understanding Overfitting in Reweighting Algorithms for Worst-group Performance
Prior work has proposed various reweighting algorithms to improve the worst-group performance of machine learning models for fairness. However, Sagawa et al. (2020) empirically found that these algorithms overfit easily …
Data AugmentationFairnessA Latent Variable Model Approach to PMI-based Word Embeddings
Semantic word embeddings represent the meaning of a word via a vector, and are created by diverse methods. Many use nonlinear operations on co-occurrence statistics, and have hand-tuned hyperparameters and reweighting me…
Word Embeddings