Infusing Prompts with Syntax and Semantics
Despite impressive success, language models often generate outputs with flawed linguistic structure. We analyze the effect of directly infusing various kinds of syntactic and semantic information into large language models. To demonstrate the value of our proposals, we focus on the translation of natural language queries to SQL, in particular dealing with languages with less resources than English, to better investigate how much help we can get from low cost syntactic and semantic information. We show that linguistic analysis can significantly boost language models, to the point that we have surpassed previous best systems.
Code (0)
등록된 구현이 없습니다.
Tasks
Natural Language QueriesTranslationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Dissecting Paraphrases: The Impact of Prompt Syntax and supplementary Information on Knowledge Retrieval from Pretrained Language Models
Pre-trained Language Models (PLMs) are known to contain various kinds of knowledge. One method to infer relational knowledge is through the use of cloze-style prompts, where a model is tasked to predict missing subjects …
RetrievalAddressing LLM Diversity by Infusing Random Concepts
Large language models (LLMs) are known to produce outputs with limited diversity. In this work, we study whether infusing random concepts in the prompts can improve the diversity of the generated outputs. To benchmark th…
Syntax-Enhanced Pre-trained Model
We study the problem of leveraging the syntactic structure of text to enhance pre-trained models such as BERT and RoBERTa. Existing methods utilize syntax of text either in the pre-training stage or in the fine-tuning st…
Entity TypingmodelQuestion AnsweringRelation ClassificationInfusing Finetuning with Semantic Dependencies
For natural language processing systems, two kinds of evidence support the use of text representations from neural language models "pretrained" on large unannotated corpora: performance on application-inspired benchmarks…
Natural Language UnderstandingUnveiling Code Pre-Trained Models: Investigating Syntax and Semantics Capacities
Past research has examined how well these models grasp code syntax, yet their understanding of code semantics still needs to be explored. We extensively analyze seven code models to investigate how code models represent …
Code CompletionCode SearchLanguage ModellingProgram Synthesis