Knowledge Prompts: Injecting World Knowledge into Language Models through Soft Prompts
Soft prompts have been recently proposed as a tool for adapting large frozen language models (LMs) to new tasks. In this work, we repurpose soft prompts to the task of injecting world knowledge into LMs. We introduce a method to train soft prompts via self-supervised learning on data from knowledge bases. The resulting soft knowledge prompts (KPs) are task independent and work as an external memory of the LMs. We perform qualitative and quantitative experiments and demonstrate that: (1) KPs can effectively model the structure of the training data; (2) KPs can be used to improve the performance of LMs in different knowledge intensive tasks.
Code (0)
등록된 구현이 없습니다.
Tasks
Self-Supervised LearningWorld KnowledgeSimilar Papers 제목 키워드 기반
Learning and Unlearning of Fabricated Knowledge in Language Models
What happens when a new piece of knowledge is introduced into the training data and how long does it last while a large language model (LM) continues to train? We investigate this question by injecting facts into LMs fro…
Data PoisoningLanguage ModelingLanguage ModellingLarge Language Model+1Search Beyond What Can Be Taught: Evolving the Knowledge Boundary in Agentic Visual Generation
Visual generators excel at rendering, but they confidently fabricate what they do not know. User requests are unbounded, evolving, and deeply long-tailed: new characters, trending entities, post-cutoff events, and more. …
Enhancing Question Answering by Injecting Ontological Knowledge through Regularization
Deep neural networks have demonstrated high performance on many natural language processing (NLP) tasks that can be answered directly from text, and have struggled to solve NLP tasks requiring external (e.g., world) know…
Question AnsweringSemantic CompositionWorld KnowledgeResearch on World Models Is Not Merely Injecting World Knowledge into Specific Tasks
World models have emerged as a critical frontier in AI research, aiming to enhance large models by infusing them with physical dynamics and world knowledge. The core objective is to enable agents to understand, predict, …
KORE: Enhancing Knowledge Injection for Large Multimodal Models via Knowledge-Oriented Controls
Large Multimodal Models encode extensive factual knowledge in their pre-trained weights. However, its knowledge remains static and limited, unable to keep pace with real-world developments, which hinders continuous knowl…
Data Augmentation