Encoders Help You Disambiguate Word Senses in Neural Machine Translation
Neural machine translation (NMT) has achieved new state-of-the-art performance in translating ambiguous words. However, it is still unclear which component dominates the process of disambiguation. In this paper, we explore the ability of NMT encoders and decoders to disambiguate word senses by evaluating hidden states and investigating the distributions of self-attention. We train a classifier to predict whether a translation is correct given the representation of an ambiguous noun. We find that encoder hidden states outperform word embeddings significantly which indicates that encoders adequately encode relevant information for disambiguation into hidden states. Decoders could provide further relevant information for disambiguation. Moreover, the attention weights and attention entropy show that self-attention can detect ambiguous nouns and distribute more attention to the context. Note that this is a revised version. The content related to decoder hidden states has been updated.
Code (0)
등록된 구현이 없습니다.
Tasks
DecoderMachine TranslationNMTTranslationWord EmbeddingsSimilar Papers 제목 키워드 기반
Moving Down the Long Tail of Word Sense Disambiguation with Gloss-Informed Biencoders
A major obstacle in Word Sense Disambiguation (WSD) is that word senses are not uniformly distributed, causing existing models to generally perform poorly on senses that are either rare or unseen during training. We prop…
Word Sense DisambiguationMoving Down the Long Tail of Word Sense Disambiguation with Gloss Informed Bi-encoders
A major obstacle in Word Sense Disambiguation (WSD) is that word senses are not uniformly distributed, causing existing models to generally perform poorly on senses that are either rare or unseen during training. We prop…
Word Sense DisambiguationLanguageNet: Learning to Find Sense Relevant Example Sentences
In this paper, we present a system, LanguageNet, which can help second language learners to search for different meanings and usages of a word. We disambiguate word senses based on the pairs of an English word and its co…
Word AlignmentA Differentiable Self-disambiguated Sense Embedding Model via Scaled Gumbel Softmax
We present a differentiable multi-prototype word representation model that disentangles senses of polysemous words and produces meaningful sense-specific embeddings without external resources. It jointly learns how to di…
Hard AttentionSentenceWord Similaritysense2vec - A Fast and Accurate Method for Word Sense Disambiguation In Neural Word Embeddings
Neural word representations have proven useful in Natural Language Processing (NLP) tasks due to their ability to efficiently model complex semantic and syntactic word relationships. However, most techniques model only o…
Dependency ParsingWord EmbeddingsWord Sense Disambiguation