Decoder-based Sense Knowledge Distillation
Large language models (LLMs) learn contextual embeddings that capture rich semantic information, yet they often overlook structured lexical knowledge such as word senses and relationships. Prior work has shown that incorporating sense dictionaries can improve knowledge distillation for encoder models, but their application to decoder as generative models remains challenging. In this paper, we introduce Decoder-based Sense Knowledge Distillation (DSKD), a framework that integrates lexical resources into the training of decoder-style LLMs without requiring dictionary lookup at inference time. Extensive experiments on diverse benchmarks demonstrate that DSKD significantly enhances knowledge distillation performance for decoders, enabling generative models to inherit structured semantics while maintaining efficient training.
Code (0)
등록된 구현이 없습니다.
Tasks
Knowledge DistillationSimilar Papers 제목 키워드 기반
Symbolic Knowledge Distillation: from General Language Models to Commonsense Models
The common practice for training commonsense models has gone from-human-to-corpus-to-machine: humans author commonsense knowledge graphs in order to train commonsense models. In this work, we investigate an alternative, …
Knowledge DistillationKnowledge GraphsLanguage ModelingLanguage Modelling+1Active Object Detection with Knowledge Aggregation and Distillation from Large Models
Accurately detecting active objects undergoing state changes is essential for comprehending human interactions and facilitating decision-making. The existing methods for active object detection (AOD) primarily rely on vi…
Active Object DetectionDecision MakingDecoderKnowledge Distillation+3I2D2: Inductive Knowledge Distillation with NeuroLogic and Self-Imitation
Commonsense capabilities of pre-trained language models dramatically improve with scale, leading many to believe that scale is the only winning recipe. But is it? Here, we investigate an alternative that a priori seems i…
Imitation LearningKnowledge DistillationMultimodal Commonsense Knowledge Distillation for Visual Question Answering
Existing Multimodal Large Language Models (MLLMs) and Visual Language Pretrained Models (VLPMs) have shown remarkable performances in the general Visual Question Answering (VQA). However, these models struggle with VQA q…
Knowledge DistillationQuestion AnsweringVisual Question AnsweringVisual Question Answering (VQA)