The Fine-Tuning Trap: Evaluating Negative Transfer and the Role of PEFT in Sub-1B Mathematical Reasoning

2026-06-05 · Rahul Nair, Chun Tao arxiv

Deploying Small Language Models (SLMs) on edge devices requires efficient fine-tuning strategies that adapt models to new tasks without degrading their general capabilities. In this study, we benchmark five sub-1B models (135M-1B) on mathematical reasoning tasks and uncover a critical vulnerability: Full Fine-Tuning (Full FT) actively harms performance in models under 300M parameters, often dropping accuracy below zero-shot baselines. This "negative transfer" makes Parameter-Efficient Fine-Tuning (PEFT) not just an efficiency preference, but a stability requirement. We find that while Low-Rank Adaptation (LoRA) and Weight-Decomposed LoRA (DoRA) perform comparably, their strengths vary by task; DoRA excels in complex reasoning (GSM8K), while LoRA dominates pattern matching (OrcaMath). In particular, Full FT is outperformed by LoRA on aligned models (Qwen2.5-0.5B) and even by simple 5-shot In-Context Learning on the smallest architectures (SmolLM2-135M). Based on these findings, we recommend defaulting to PEFT for all aligned sub-1B models and caution against Full FT for any architecture smaller than 500M parameters to prevent catastrophic forgetting. Reproduction of this work can be found at https://github.com/gulguluu/tiny-slm-finetune-compare.

📄 PDF Abstract BibTeX arXiv:2606.06920

Code (0)

등록된 구현이 없습니다.

Tasks

parameter-efficient fine-tuning Mathematical Reasoning

Catastrophic Forgetting Meets Negative Transfer: Batch Spectral Shrinkage for Safe Transfer Learning

2019-12-01 · NeurIPS 2019 12 · Xinyang Chen, Sinan Wang, Bo Fu, Mingsheng Long 외

Before sufficient training data is available, fine-tuning neural networks pre-trained on large-scale datasets substantially outperforms training from random initialization. However, fine-tuning methods suffer from two di…

Transfer Learning

Transfer Risk Map: Mitigating Pixel-level Negative Transfer in Medical Segmentation

2025-02-04 · Shutong Duan, Jingyun Yang, Yang Tan, Guoqing Zhang 외

How to mitigate negative transfer in transfer learning is a long-standing and challenging issue, especially in the application of medical image segmentation. Existing methods for reducing negative transfer focus on class…

Brain Segmentation Image Segmentation Medical Image Segmentation Segmentation+2

Concept-wise Fine-tuning Matters in Preventing Negative Transfer

2023-11-12 · ICCV 2023 1 · Yunqiao Yang, Long-Kai Huang, Ying WEI

A multitude of prevalent pre-trained models mark a major milestone in the development of artificial intelligence, while fine-tuning has been a common practice that enables pretrained models to figure prominently in a wid…

Data Synthesis and Parameter-Efficient Fine-Tuning for Low-Resource NMT: A Case Study on Q'eqchi' Mayan

2026-06-08 · Alexander Chulzhanov, Soeren Eberhardt, Arjun Mukherjee arxiv

Neural machine translation for digitally low-resource Indigenous languages is often hindered by extreme data scarcity, prompting reliance on extractive web-scraping. To ensure data sovereignty, this study introduces a da…

parameter-efficient fine-tuning Machine Translation Multi-Task Learning

Fine-Tuning Language Models on Multiple Datasets for Citation Intention Classification

2024-10-17 · Zeren Shui, Petros Karypis, Daniel S. Karls, Mingjian Wen 외

Citation intention Classification (CIC) tools classify citations by their intention (e.g., background, motivation) and assist readers in evaluating the contribution of scientific literature. Prior research has shown that…

Multi-Task Learning