Slimming Down LLMs Without Losing Their Minds
This paper investigates and validates the impact of fine-tuning on large language model performance, focusing on parameter-efficient methods (LoRA and QLoRA). We evaluate model capabilities across three key domains: (1) commonsense reasoning (HellaSwag), (2) mathematical reasoning (GSM8K), and (3) multi-domain knowledge (MMLU-CS). Our findings demonstrate that: (1) LoRA-based methods effectively improve task-specific performance while maintaining computational efficiency, and (2) performance strongly depends on alignment between fine-tuning dataset and benchmark tasks. The study provides both theoretical insights into parameter-efficient mechanisms and practical guidance for developers implementing efficient LLM adaptation with limited resources.
Code (0)
등록된 구현이 없습니다.
Tasks
Computational EfficiencyGSM8KHellaSwagLanguage ModelingLanguage ModellingLarge Language ModelMathematical ReasoningMMLUSimilar Papers 제목 키워드 기반
DocKylin: A Large Multimodal Model for Visual Document Understanding with Efficient Visual Slimming
Current multimodal large language models (MLLMs) face significant challenges in visual document understanding (VDU) tasks due to the high resolution, dense text, and complex layouts typical of document images. These char…
document understandingPatch Slimming for Efficient Vision Transformers
This paper studies the efficiency problem for visual transformers by excavating redundant calculation in given networks. The recent transformer architecture has demonstrated its effectiveness for achieving excellent perf…
Efficient ViTsDANCE: DAta-Network Co-optimization for Efficient Segmentation Model Training and Inference
Semantic segmentation for scene understanding is nowadays widely demanded, raising significant challenges for the algorithm efficiency, especially its applications on resource-limited platforms. Current segmentation mode…
Scene UnderstandingSegmentationSemantic SegmentationSoLA: Leveraging Soft Activation Sparsity and Low-Rank Decomposition for Large Language Model Compression
Large language models (LLMs) have demonstrated impressive capabilities across various tasks, but the billion-scale parameters pose deployment challenges. Although existing methods attempt to reduce the scale of LLMs, the…
Model CompressionLearning Efficient Convolutional Networks through Network Slimming
The deployment of deep convolutional neural networks (CNNs) in many real world applications is largely hindered by their high computational cost. In this paper, we propose a novel learning scheme for CNNs to simultaneous…
image-classificationImage ClassificationNeural Architecture Search