Language Modeling with Syntactic and Semantic Representation for Sentence Acceptability Predictions
In this paper, we investigate the effect of enhancing lexical embeddings in LSTM language models (LM) with syntactic and semantic representations. We evaluate the language models using perplexity, and we evaluate the performance of the models on the task of predicting human sentence acceptability judgments. We train LSTM language models on sentences automatically annotated with universal syntactic dependency roles (Nivre, 2016), dependency depth and universal semantic tags (Abzianidze et al., 2017) to predict sentence acceptability judgments. Our experiments indicate that syntactic tags lower perplexity, while semantic tags increase it. Our experiments also show that neither syntactic nor semantic tags improve the performance of LSTM language models on the task of predicting sentence acceptability judgments.
Code (1)
Tasks
Language ModelingLanguage ModellingSentenceSimilar Papers 제목 키워드 기반
On the Relationship between Sentence Analogy Identification and Sentence Structure Encoding in Large Language Models
The ability of Large Language Models (LLMs) to encode syntactic and semantic structures of language is well examined in NLP. Additionally, analogy identification, in the form of word analogies are extensively studied in …
Language ModelingLanguage ModellingSentenceDifferential syntactic and semantic encoding in LLMs
We study how syntactic and semantic information is encoded in inner layer representations of Large Language Models (LLMs), focusing on the very large DeepSeek-V3. We find that, by averaging hidden-representation vectors …
Disentangling Semantics and Syntax in Sentence Embeddings with Pre-trained Language Models
Pre-trained language models have achieved huge success on a wide range of NLP tasks. However, contextual representations from pre-trained models contain entangled semantic and syntactic information, and therefore cannot …
Semantic SimilaritySemantic Textual SimilaritySentenceSentence Embedding+2Recursive Tree Attention: Improving Semantic Representations with Syntactic Tree Structured Attention Mechanism
Attention mechanism has shown its effectiveness in state-of-the-art methods on various tasks in natural language processing (NLP). However, these methods are still using attention mechanism in plain, linear topological s…
Constituency ParsingRepresentation LearningSentencetext-classification+1Fake Sentence Detection as a Training Task for Sentence Encoding
Sentence encoders are typically trained on language modeling tasks which enable them to use large unlabeled datasets. While these models achieve state-of-the-art results on many sentence-level tasks, they are difficult …
Binary ClassificationLanguage ModelingLanguage ModellingSentence