KBLaM: Knowledge Base augmented Language Model
In this paper, we propose Knowledge Base augmented Language Model (KBLaM), a new method for augmenting Large Language Models (LLMs) with external knowledge. KBLaM works with a knowledge base (KB) constructed from a corpus of documents, transforming each piece of knowledge in the KB into continuous key-value vector pairs via pre-trained sentence encoders with linear adapters and integrating them into pre-trained LLMs via a specialized rectangular attention mechanism. Unlike Retrieval-Augmented Generation, KBLaM eliminates external retrieval modules, and unlike in-context learning, its computational overhead scales linearly with KB size rather than quadratically. Our approach enables integrating a large KB of more than 10K triples into an 8B pre-trained LLM of only 8K context window on one single A100 80GB GPU and allows for dynamic updates without model fine-tuning or retraining. Experiments demonstrate KBLaM's effectiveness in various tasks, including question-answering and open-ended reasoning, while providing interpretable insights into its use of the augmented knowledge.
Code (1)
Tasks
8kGPUIn-Context LearningLanguage ModelingLanguage ModellingmodelQuestion AnsweringRetrievalRetrieval-augmented GenerationSentenceMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Scalable and Explainable Enterprise Knowledge Discovery Using Graph-Centric Hybrid Retrieval
Modern enterprises manage vast knowledge distributed across heterogeneous systems such as Jira, Git repositories, Confluence, and wikis. Conventional retrieval methods based on keyword search or static embeddings often f…
Semantic SimilarityRetrieval-based Knowledge Augmented Vision Language Pre-training
With the recent progress in large-scale vision and language representation learning, Vision Language Pre-training (VLP) models have achieved promising improvements on various multi-modal downstream tasks. Albeit powerful…
Entity LinkingKnowledge GraphsRelational ReasoningRepresentation Learning+3CroSearch-R1: Better Leveraging Cross-lingual Knowledge for Retrieval-Augmented Generation
A multilingual collection may contain useful knowledge in other languages to supplement and correct the facts in the original language for Retrieval-Augmented Generation (RAG). However, the vanilla approach that simply c…
Reinforcement LearningRetrieval-Augmented Machine Translation with Unstructured Knowledge
Retrieval-augmented generation (RAG) introduces additional information to enhance large language models (LLMs). In machine translation (MT), previous work typically retrieves in-context examples from paired MT corpora, o…
Knowledge GraphsMachine TranslationRAGRetrieval+3Commonsense Knowledge-Augmented Pretrained Language Models for Causal Reasoning Classification
Commonsense knowledge can be leveraged for identifying causal relations in text. In this work, we convert triples in ATOMIC2020, a wide coverage commonsense reasoning knowledge graph, to natural language text and continu…
ClassificationCommonsense Causal ReasoningLanguage ModelingLanguage Modelling