Deep Neural Networks at the Service of Multilingual Parallel Sentence Extraction
Wikipedia provides an invaluable source of parallel multilingual data, which are in high demand for various sorts of linguistic inquiry, including both theoretical and practical studies. We introduce a novel end-to-end neural model for large-scale parallel data harvesting from Wikipedia. Our model is language-independent, robust, and highly scalable. We use our system for collecting parallel German-English, French-English and Persian-English sentences. Human evaluations at the end show the strong performance of this model in collecting high-quality parallel data. We also propose a statistical framework which extends the results of our human evaluation to other language pairs. Our model also obtained a state-of-the-art result on the German-English dataset of BUCC 2017 shared task on parallel sentence extraction from comparable corpora.
Code (0)
등록된 구현이 없습니다.
Tasks
Information RetrievalMachine TranslationSentenceSimilar Papers 제목 키워드 기반
A Deep Neural Network Approach To Parallel Sentence Extraction
Parallel sentence extraction is a task addressing the data sparsity problem found in multilingual natural language processing applications. We propose an end-to-end deep neural network approach to detect translational eq…
Feature EngineeringMachine TranslationSentenceTranslation+1A Multilingual Dataset for Evaluating Parallel Sentence Extraction from Comparable Corpora
Bilingual Text Extraction as Reading Comprehension
In this paper, we propose a method to extract bilingual texts automatically from noisy parallel corpora by framing the problem as a token-level span prediction, such as SQuAD-style Reading Comprehension. To extract a spa…
ArticlesReading ComprehensionSentenceTranslationExtracting Parallel Sentences with Bidirectional Recurrent Neural Networks to Improve Machine Translation
Parallel sentence extraction is a task addressing the data sparsity problem found in multilingual natural language processing applications. We propose a bidirectional recurrent neural network based approach to extract pa…
ArticlesFeature EngineeringMachine TranslationSentence+1Multilingual Sentence Transformer as A Multilingual Word Aligner
Multilingual pretrained language models (mPLMs) have shown their effectiveness in multilingual word alignment induction. However, these methods usually start from mBERT or XLM-R. In this paper, we investigate whether mul…
SentenceWord AlignmentXLM-R