Putting words in context: LSTM language models and lexical ambiguity
In neural network models of language, words are commonly represented using context-invariant representations (word embeddings) which are then put in context in the hidden layers. Since words are often ambiguous, representing the contextually relevant information is not trivial. We investigate how an LSTM language model deals with lexical ambiguity in English, designing a method to probe its hidden representations for lexical and contextual information about words. We find that both types of information are represented to a large extent, but also that there is room for improvement for contextual information.
Code (1)
Tasks
Language ModelingLanguage ModellingWord EmbeddingsMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Fast and Accurate Recurrent Neural Network Acoustic Models for Speech Recognition
We have recently shown that deep Long Short-Term Memory (LSTM) recurrent neural networks (RNNs) outperform feed forward deep neural networks (DNNs) as acoustic models for speech recognition. More recently, we have shown …
General Classificationspeech-recognitionSpeech RecognitionHead-Lexicalized Bidirectional Tree LSTMs
Sequential LSTMs have been extended to model tree structures, giving competitive results for a number of tasks. Existing methods model constituent trees by bottom-up combinations of constituent nodes, making direct use o…
Language ModelingLanguage ModellingRelation ExtractionSentiment AnalysisBidirectional Tree-Structured LSTM with Head Lexicalization
Sequential LSTM has been extended to model tree structures, giving competitive results for a number of tasks. Existing methods model constituent trees by bottom-up combinations of constituent nodes, making direct use of …
General ClassificationLearning to Represent Words in Context with Multilingual Supervision
We present a neural network architecture based on bidirectional LSTMs to compute representations of words in the sentential contexts. These context-sensitive word representations are suitable for, e.g., distinguishing di…
Machine TranslationTranslationSemantic Modelling with Long-Short-Term Memory for Information Retrieval
In this paper we address the following problem in web document and information retrieval (IR): How can we use long-term context information to gain better IR performance? Unlike common IR methods that use bag of words re…
Information RetrievalRetrieval