Knowledge Distillation for Large Language Models
We propose a resource-efficient framework for compressing large language models through knowledge distillation, combined with guided chain-of-thought reinforcement learning. Using Qwen 3B as the teacher and Qwen 0.5B as the student, we apply knowledge distillation across English Dolly-15k, Spanish Dolly-15k, and code BugNet and PyTorrent datasets, with hyperparameters tuned in the English setting to optimize student performance. Across tasks, the distilled student retains a substantial portion of the teacher's capability while remaining significantly smaller: 70% to 91% in English, up to 95% in Spanish, and up to 93.5% Rouge-L in code. For coding tasks, integrating chain-of-thought prompting with Group Relative Policy Optimization using CoT-annotated Codeforces data improves reasoning coherence and solution correctness compared to knowledge distillation alone. Post-training 4-bit weight quantization further reduces memory footprint and inference latency. These results show that knowledge distillation combined with chain-of-thought guided reinforcement learning can produce compact, efficient models suitable for deployment in resource-constrained settings.
Code (0)
등록된 구현이 없습니다.
Tasks
Knowledge DistillationReinforcement LearningSimilar Papers 제목 키워드 기반
GKD: A General Knowledge Distillation Framework for Large-scale Pre-trained Language Model
Currently, the reduction in the parameter scale of large-scale pre-trained language models (PLMs) through knowledge distillation has greatly facilitated their widespread deployment on various devices. However, the deploy…
General KnowledgeKnowledge DistillationLanguage ModelingLanguage ModellingWhy Knowledge Distillation Amplifies Gender Bias and How to Mitigate from the Perspective of DistilBERT
Knowledge distillation is widely used to transfer the language understanding of a large model to a smaller model.However, after knowledge distillation, it was found that the smaller model is more biased by gender compare…
Knowledge DistillationAME: Aligned Manifold Entropy for Robust Vision-Language Distillation
Knowledge distillation is a long-established technique for knowledge transfer, and has regained attention in the context of the recent emergence of large vision-language models (VLMs). However, vision-language knowledge …
Knowledge DistillationDDK: Distilling Domain Knowledge for Efficient Large Language Models
Despite the advanced intelligence abilities of large language models (LLMs) in various applications, they still face significant computational and storage demands. Knowledge Distillation (KD) has emerged as an effective …
Knowledge DistillationFeature Alignment and Representation Transfer in Knowledge Distillation for Large Language Models
Knowledge distillation (KD) is a technique for transferring knowledge from complex teacher models to simpler student models, significantly enhancing model efficiency and accuracy. It has demonstrated substantial advancem…
image-classificationImage ClassificationKnowledge DistillationLanguage Modeling+7