Regressing Word and Sentence Embeddings for Regularization of Neural Machine Translation
In recent years, neural machine translation (NMT) has become the dominant approach in automated translation. However, like many other deep learning approaches, NMT suffers from overfitting when the amount of training data is limited. This is a serious issue for low-resource language pairs and many specialized translation domains that are inherently limited in the amount of available supervised data. For this reason, in this paper we propose regressing word (ReWE) and sentence (ReSE) embeddings at training time as a way to regularize NMT models and improve their generalization. During training, our models are trained to jointly predict categorical (words in the vocabulary) and continuous (word and sentence embeddings) outputs. An extensive set of experiments over four language pairs of variable training set size has showed that ReWE and ReSE can outperform strong state-of-the-art baseline models, with an improvement that is larger for smaller training sets (e.g., up to +5:15 BLEU points in Basque-English translation). Visualizations of the decoder's output space show that the proposed regularizers improve the clustering of unique words, facilitating correct predictions. In a final experiment on unsupervised NMT, we show that ReWE and ReSE are also able to improve the quality of machine translation when no parallel data are available.
Code (0)
등록된 구현이 없습니다.
Tasks
ClusteringMachine TranslationNMTSentenceSentence EmbeddingsTranslationSimilar Papers 제목 키워드 기반
ReWE: Regressing Word Embeddings for Regularization of Neural Machine Translation Systems
Regularization of neural machine translation is still a significant problem, especially in low-resource settings. To mollify this problem, we propose regressing word embeddings (ReWE) as a new regularization technique in…
Machine TranslationTranslationWord EmbeddingsSentence Curve Language Models
Language models (LMs) are a central component of modern AI systems, and diffusion language models (DLMs) have recently emerged as a competitive alternative. Both paradigms rely on word embeddings not only to represent th…
Knowledge DistillationMGNC-CNN: A Simple Approach to Exploiting Multiple Word Embeddings for Sentence Classification
We introduce a novel, simple convolution neural network (CNN) architecture - multi-group norm constraint CNN (MGNC-CNN) that capitalizes on multiple sets of word embeddings for sentence classification. MGNC-CNN extracts …
General ClassificationSentenceSentence ClassificationWord EmbeddingsMaxMatch-Dropout: Subword Regularization for WordPiece
We present a subword regularization method for WordPiece, which uses a maximum matching algorithm for tokenization. The proposed method, MaxMatch-Dropout, randomly drops words in a search using the maximum matching algor…
Machine TranslationText ClassificationTranslationSentiment Analysis of Citations Using Word2vec
Citation sentiment analysis is an important task in scientific paper analysis. Existing machine learning techniques for citation sentiment analysis are focusing on labor-intensive feature engineering, which requires larg…
Feature EngineeringGeneral ClassificationSentenceSentiment Analysis+1