Encodings of Source Syntax: Similarities in NMT Representations Across Target Languages
We train neural machine translation (NMT) models from English to six target languages, using NMT encoder representations to predict ancestor constituent labels of source language words. We find that NMT encoders learn similar source syntax regardless of NMT target language, relying on explicit morphosyntactic cues to extract syntactic features from source sentences. Furthermore, the NMT encoders outperform RNNs trained directly on several of the constituent label prediction tasks, suggesting that NMT encoder representations can be used effectively for natural language tasks involving syntax. However, both the NMT encoders and the directly-trained RNNs learn substantially different syntactic information from a probabilistic context-free grammar (PCFG) parser. Despite lower overall accuracy scores, the PCFG often performs well on sentences for which the RNN-based models perform poorly, suggesting that RNN architectures are constrained in the types of syntax they can learn.
Code (0)
등록된 구현이 없습니다.
Tasks
Machine TranslationNMTTranslationSimilar Papers 제목 키워드 기반
Assessment of Pre-Trained Models Across Languages and Grammars
We present an approach for assessing how multilingual large language models (LLMs) learn syntax in terms of multi-formalism syntactic structures. We aim to recover constituent and dependency structures by casting parsing…
Dependency ParsingWhen Does Syntax Mediate Neural Language Model Performance? Evidence from Dropout Probes
Recent causal probing literature reveals when language models and syntactic probes use similar representations. Such techniques may yield "false negative" causality results: models may use representations of syntax, but …
Language ModelingLanguage ModellingWhen Does Syntax Mediate Neural Language Model Performance? Evidence from Dropout Probes
Recent causal probing literature reveals when language models and syntactic probes use similar representations. Such techniques may yield ``false negative'' causality results: models may use representations of syntax, bu…
Language ModelingLanguage ModellingIR2Vec: LLVM IR based Scalable Program Embeddings
We propose IR2Vec, a Concise and Scalable encoding infrastructure to represent programs as a distributed embedding in continuous space. This distributed embedding is obtained by combining representation learning methods …
Representation LearningMachine Translation by Projecting Text into the Same Phonetic-Orthographic Space Using a Common Encoding
The use of subword embedding has proved to be a major innovation in Neural Machine Translation (NMT). It helps NMT to learn better context vectors for Low Resource Languages (LRLs) so as to predict the target words by be…
Machine TranslationNMTTranslation