KnowLA: Enhancing Parameter-efficient Finetuning with Knowledgeable Adaptation
Parameter-efficient finetuning (PEFT) is a key technique for adapting large language models (LLMs) to downstream tasks. In this paper, we study leveraging knowledge graph embeddings to improve the effectiveness of PEFT. We propose a knowledgeable adaptation method called KnowLA. It inserts an adaptation layer into an LLM to integrate the embeddings of entities appearing in the input text. The adaptation layer is trained in combination with LoRA on instruction data. Experiments on six benchmarks with two popular LLMs and three knowledge graphs demonstrate the effectiveness and robustness of KnowLA. We show that \modelname can help activate the relevant parameterized knowledge in an LLM to answer a question without changing its parameters or input prompts.
Code (1)
Tasks
Knowledge Graph EmbeddingsKnowledge GraphsSimilar Papers 제목 키워드 기반
Parameter Efficient Finetuning for Speech Emotion Recognition and Domain Adaptation
Foundation models have shown superior performance for speech emotion recognition (SER). However, given the limited data in emotion corpora, finetuning all parameters of large pre-trained models for SER can be both resour…
Cross-corpusDomain AdaptationEmotion RecognitionSpeech Emotion RecognitionDeLoRA: Decoupling Angles and Strength in Low-rank Adaptation
Parameter-Efficient FineTuning (PEFT) methods have recently gained significant popularity thanks to the widespread availability of large-scale pretrained models. These methods allow for quick adaptation to downstream tas…
Image GenerationNatural Language UnderstandingRobust Few-Shot Vision-Language Model Adaptation
Pretrained VLMs achieve strong performance on downstream tasks when adapted with just a few labeled examples. As the adapted models inevitably encounter out-of-distribution (OOD) test data that deviates from the in-distr…
Language ModelingLanguage ModellingmodelRetrievalMulti-Stage Prompting for Knowledgeable Dialogue Generation
Existing knowledge-grounded dialogue systems typically use finetuned versions of a pretrained language model (LM) and large-scale knowledge bases. These models typically fail to generalize on topics outside of the knowle…
Dialogue GenerationLanguage ModellingRetrievalMulti-Stage Prompting for Knowledgeable Dialogue Generation
Existing knowledge-grounded dialogue systems typically use finetuned versions of a pretrained language model (LM) and large-scale knowledge bases. These models typically fail to generalize on topics outside of the knowle…
Dialogue GenerationLanguage ModellingRetrieval