OpenGrok: Enhancing SNS Data Processing with Distilled Knowledge and Mask-like Mechanisms
This report details Lumen Labs' novel approach to processing Social Networking Service (SNS) data. We leverage knowledge distillation, specifically a simple distillation method inspired by DeepSeek-R1's CoT acquisition, combined with prompt hacking, to extract valuable training data from the Grok model. This data is then used to fine-tune a Phi-3-mini model, augmented with a mask-like mechanism specifically designed for handling the nuances of SNS data. Our method demonstrates state-of-the-art (SOTA) performance on several SNS data processing tasks, outperforming existing models like Grok, Phi-3, and GPT-4. We provide a comprehensive analysis of our approach, including mathematical formulations, engineering details, ablation studies, and comparative evaluations.
Code (1)
Tasks
Knowledge DistillationMMLUText-To-SQLMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Enhancing deep learning models for time series classification via knowledge distillation
Deep learning has achieved remarkable success in various domains including time series analysis, computer vision and natural language processing. However, high computational and memory demands of state-of-the-art archite…
Time Series ClassificationKnowledge DistillationTime Series AnalysisMulti-Stage Balanced Distillation: Addressing Long-Tail Challenges in Sequence-Level Knowledge Distillation
Large language models (LLMs) have significantly advanced various natural language processing tasks, but deploying them remains computationally expensive. Knowledge distillation (KD) is a promising solution, enabling the …
Knowledge DistillationEnhancing Knowledge Distillation for LLMs with Response-Priming Prompting
Large language models (LLMs) have demonstrated remarkable performance across a wide range of natural language processing (NLP) tasks. However, these models are often difficult to deploy due to significant computational r…
GSM8KKnowledge DistillationTransfer LearningImproving generalizability of distilled self-supervised speech processing models under distorted settings
Self-supervised learned (SSL) speech pre-trained models perform well across various speech processing tasks. Distilled versions of SSL models have been developed to match the needs of on-device speech applications. Thoug…
Knowledge DistillationDistilQwen2.5: Industrial Practices of Training Distilled Open Lightweight Language Models
Enhancing computational efficiency and reducing deployment costs for large language models (LLMs) have become critical challenges in various resource-constrained scenarios. In this work, we present DistilQwen2.5, a famil…
Computational EfficiencyInstruction Following