Natural Language Multitasking: Analyzing and Improving Syntactic Saliency of Hidden Representations
We train multi-task autoencoders on linguistic tasks and analyze the learned hidden sentence representations. The representations change significantly when translation and part-of-speech decoders are added. The more decoders a model employs, the better it clusters sentences according to their syntactic similarity, as the representation space becomes less entangled. We explore the structure of the representation space by interpolating between sentences, which yields interesting pseudo-English sentences, many of which have recognizable syntactic structure. Lastly, we point out an interesting property of our models: The difference-vector between two sentences can be added to change a third sentence with similar features in a meaningful way.
Code (0)
등록된 구현이 없습니다.
Tasks
SentenceTranslationSimilar Papers 제목 키워드 기반
Multitasking Models are Robust to Structural Failure: A Neural Model for Bilingual Cognitive Reserve
We find a surprising connection between multitask learning and robustness to neuron failures. Our experiments show that bilingual language models retain higher performance under various neuron perturbations, such as rand…
Representation LearningA Novel Method to Study Bottom-up Visual Saliency and its Neural Mechanism
In this study, we propose a novel method to measure bottom-up saliency maps of natural images. In order to eliminate the influence of top-down signals, backward masking is used to make stimuli (natural images) subjective…
Towards Explainability in NLP: Analyzing and Calculating Word Saliency through Word Properties
The wide use of black-box models in natural language processing brings great challenges to the understanding of the decision basis, the trustworthiness of the prediction results, and the improvement of the model performa…
Better Character Language Modeling Through Morphology
We incorporate morphological supervision into character language models (CLMs) via multitasking and show that this addition improves bits-per-character (BPC) performance across 24 languages, even when the morphology data…
Language ModelingLanguage ModellingLearning Music Helps You Read: Using Transfer to Study Linguistic Structure in Language Models
We propose transfer learning as a method for analyzing the encoding of grammatical structure in neural language models. We train LSTMs on non-linguistic data and evaluate their performance on natural language to assess w…
Language AcquisitionLanguage ModellingTransfer Learning