Accessing Higher Dimensions for Unsupervised Word Translation
The striking ability of unsupervised word translation has been demonstrated recently with the help of low-dimensional word vectors / pretraining, which is used by all successful methods and assumed to be necessary. We test and challenge this assumption by developing a method that can also make use of high dimensional signal. Freed from the limits of low dimensions, we show that relying on low-dimensional vectors and their incidental properties miss out on better denoising methods and signals in high dimensions, thus stunting the potential of the data. Our results show that unsupervised translation can be achieved more easily and robustly than previously thought -- less than 80MB and minutes of CPU time is required to achieve over 50\% accuracy for English to Finnish, Hungarian, and Chinese translations when trained in the same domain; even under domain mismatch, the method still works fully unsupervised on English NewsCrawl to Chinese Wikipedia and English Europarl to Spanish Wikipedia, among others. These results challenge prevailing assumptions on the necessity and superiority of low-dimensional vectors and show that the higher dimension signal can be used rather than thrown away.
Code (1)
Similar Papers 제목 키워드 기반
Target Foresight Based Attention for Neural Machine Translation
In neural machine translation, an attention model is used to identify the aligned source words for a target word (target foresight word) in order to select translation context, but it does not make use of any information…
Language ModelingLanguage ModellingMachine TranslationTranslationUnsupervised Neural Dialect Translation with Commonality and Diversity Modeling
As a special machine translation task, dialect translation has two main characteristics: 1) lack of parallel training corpus; and 2) possessing similar grammar between two sides of the translation. In this paper, we inve…
DiversityMachine TranslationTranslationThe LMU Munich System for the WMT20 Very Low Resource Supervised MT Task
We present our systems for the WMT20 Very Low Resource MT Task for translation between German and Upper Sorbian. For training our systems, we generate synthetic data by both back- and forward-translation. Additionally, w…
Transfer LearningTranslationMultilingual Word Embeddings using Multigraphs
We present a family of neural-network--inspired models for computing continuous word representations, specifically designed to exploit both monolingual and multilingual text. This framework allows us to perform unsupervi…
Machine TranslationMultilingual Word EmbeddingsSemantic SimilaritySemantic Textual Similarity+2The LMU Munich Unsupervised Machine Translation Systems
We describe LMU Munich{'}s unsupervised machine translation systems for English↔German translation. These systems were used to participate in the WMT18 news translation shared task and more specifically, for the unsupe…
DenoisingLanguage ModelingLanguage ModellingMachine Translation+3