홈 › Papers

GIFT-SW: Gaussian noise Injected Fine-Tuning of Salient Weights for LLMs

2024-08-27 · Maxim Zhelnin, Viktor Moskvoretskii, Egor Shvetsov, Egor Venediktov, Mariya Krylova, Aleksandr Zuev, Evgeny Burnaev

Parameter Efficient Fine-Tuning (PEFT) methods have gained popularity and democratized the usage of Large Language Models (LLMs). Recent studies have shown that a small subset of weights significantly impacts performance. Based on this observation, we introduce a novel PEFT method, called Gaussian noise Injected Fine Tuning of Salient Weights (GIFT-SW). Our method updates only salient columns, while injecting Gaussian noise into non-salient ones. To identify these columns, we developeda generalized sensitivity metric that extends and unifies metrics from previous studies. Experiments with LLaMA models demonstrate that GIFT-SW outperforms full fine-tuning and modern PEFT methods under the same computational budget. Moreover, GIFT-SW offers practical advantages to recover performance of models subjected to mixed-precision quantization with keeping salient weights in full precision.

📄 PDF Abstract BibTeX arXiv:2408.15300

Code (1)

On-Point-RND/GIFT_SW 공식 구현 pytorch

Tasks

parameter-efficient fine-tuning Quantization

Methods 이 논문이 사용한 방법론

LLaMA LLaMA is a collection of foundation language models ranging from 7B to 65B parameters. It is based on the transformer architecture with various improvements that were…

GIFT: Gradient-aware Immunization of diffusion models against malicious Fine-Tuning with safe concepts retention

2025-07-18 · Amro Abdalla, Ismail Shaheen, Dan DeGenaro, Rupayan Mallick 외 arxiv

We present GIFT: a {G}radient-aware {I}mmunization technique to defend diffusion models against malicious {F}ine-{T}uning while preserving their ability to generate safe content. Existing safety mechanisms like safety ch…

GIFT: Guided Fine-Tuning and Transfer for Enhancing Instruction-Tuned Language Models

2026-05-02 · Zhiwen Ruan, Yichao Du, Jianjie Zheng, Longyue Wang 외 arxiv

A promising paradigm for adapting instruction-tuned language models is to learn task-specific updates on a pretrained base model and subsequently merge them into the instruction-tuned model. However, existing approaches …

Generative Parameter-Efficient Fine-Tuning

2023-12-01 · Chinmay Savadikar, Xi Song, Tianfu Wu

We present Generative Parameter-Efficient Fine-Tuning (GIFT) for adapting pretrained Transformer backbones on downstream tasks. GIFT learns to generate the fine-tuned weights for a layer directly from its pretrained weig…

Arithmetic Reasoning Fine-Grained Image Classification Instruction Following parameter-efficient fine-tuning+1

GIFT: Guided Importance-Aware Fine-Tuning for Diffusion Language Models

2025-09-25 · Guowei Xu, Wenxin Xu, Jiawang Zhao, Kaisheng Ma arxiv

Diffusion models have recently shown strong potential in language modeling, offering faster generation compared to traditional autoregressive approaches. However, applying supervised fine-tuning (SFT) to diffusion models…

GiFT: Gibbs Fine-Tuning for Code Generation

2025-02-17 · Haochen Li, Wanjin Feng, Xin Zhou, Zhiqi Shen

Training Large Language Models (LLMs) with synthetic data is a prevalent practice in code generation. A key approach is self-training, where LLMs are iteratively trained on self-generated correct code snippets. In this c…

Code Generation valid