RoCoFT: Efficient Finetuning of Large Language Models with Row-Column Updates

2024-10-14 · Md Kowsher, Tara Esmaeilbeig, Chun-Nam Yu, Mojtaba Soltanalian, Niloofar Yousefi

We propose RoCoFT, a parameter-efficient fine-tuning method for large-scale language models (LMs) based on updating only a few rows and columns of the weight matrices in transformers. Through extensive experiments with medium-size LMs like BERT and RoBERTa, and larger LMs like Bloom-7B, Llama2-7B, and Llama2-13B, we show that our method gives comparable or better accuracies than state-of-art PEFT methods while also being more memory and computation-efficient. We also study the reason behind the effectiveness of our method with tools from neural tangent kernel theory. We empirically demonstrate that our kernel, constructed using a restricted set of row and column parameters, are numerically close to the full-parameter kernel and gives comparable classification performance. Ablation studies are conducted to investigate the impact of different algorithmic choices, including the selection strategy for rows and columns as well as the optimal rank for effective implementation of our method.

📄 PDF Abstract BibTeX arXiv:2410.10075

Code (1)

Kowsher/RoCoFT 공식 구현 pytorch

Tasks

parameter-efficient fine-tuning

Methods 이 논문이 사용한 방법론

Refunds@Expedia|||How do I get a full refund from Expedia? “How do I get a full refund from Expedia? How do I get a full refund from Expedia? – Call ☎️ +1-(888) 829 (0881) or +1-805-330-4056 or +1-805-330-4056 for Quick Help &…

Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Multi-Head Attention 설명 없음

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

WordPiece 설명 없음

Residual Connection 설명 없음

RoseLoRA: Row and Column-wise Sparse Low-rank Adaptation of Pre-trained Language Model for Knowledge Editing and Fine-tuning

2024-06-16 · Haoyu Wang, Tianci Liu, Ruirui Li, Monica Cheng 외

Pre-trained language models, trained on large-scale corpora, demonstrate strong generalizability across various NLP tasks. Fine-tuning these models for specific tasks typically involves updating all parameters, which is …

knowledge editing Language Modeling Language Modelling parameter-efficient fine-tuning

Conda: Column-Normalized Adam for Training Large Language Models Faster

2025-09-29 · Junjie Wang, Pan Zhou, Yiming Dong, Huan Li 외 arxiv

Large language models (LLMs) have demonstrated impressive generalization and emergent capabilities, yet their pre-training remains computationally expensive and sensitive to optimization dynamics. While Adam-based optimi…

Continual Learning via Sparse Memory Finetuning

2025-10-16 · Jessy Lin, Luke Zettlemoyer, Gargi Ghosh, Wen-Tau Yih 외 arxiv

Modern language models are powerful, but typically static after deployment. A major obstacle to building models that continually learn over time is catastrophic forgetting, where updating on new data erases previously ac…

Continual Learning Question Answering

Improving Sparse Memory Finetuning

2026-04-06 · Satyam Goyal, Anirudh Kanchi, Garv Shah, Prakhar Gupta arxiv

Large Language Models (LLMs) are typically static after training, yet real-world applications require continual adaptation to new knowledge without degrading existing capabilities. Standard approaches to updating models,…

Continual Learning

Self-Adapting Language Models

2025-06-12 · Adam Zweiger, Jyothish Pari, Han Guo, Ekin Akyürek 외

Large language models (LLMs) are powerful but static; they lack mechanisms to adapt their weights in response to new tasks, knowledge, or examples. We introduce Self-Adapting LLMs (SEAL), a framework that enables LLMs to…

Data Augmentation