홈 › Papers

HaLoRA: Hardware-aware Low-Rank Adaptation for Large Language Models Based on Hybrid Compute-in-Memory Architecture

2025-02-27 · Taiqiang Wu, Chenchen Ding, Wenyong Zhou, Yuxin Cheng, Xincheng Feng, Shuqi Wang, Chufan Shi, Zhengwu Liu, Ngai Wong

Low-rank adaptation (LoRA) is a predominant parameter-efficient finetuning method to adapt large language models (LLMs) for downstream tasks. In this paper, we first propose to deploy the LoRA-finetuned LLMs on the hybrid compute-in-memory (CIM) architecture (i.e., pretrained weights onto RRAM and LoRA onto SRAM). To address performance degradation from RRAM's inherent noise, we design a novel Hardware-aware Low-rank Adaption (HaLoRA) method, aiming to train a LoRA branch that is both robust and accurate by aligning the training objectives under both ideal and noisy conditions. Experiments finetuning LLaMA 3.2 1B and 3B demonstrate HaLoRA's effectiveness across multiple reasoning tasks, achieving up to 22.7 improvement in average score while maintaining robustness at various noise levels.

📄 PDF Abstract BibTeX arXiv:2502.19747

Code (0)

등록된 구현이 없습니다.

Methods 이 논문이 사용한 방법론

LLaMA LLaMA is a collection of foundation language models ranging from 7B to 65B parameters. It is based on the transformer architecture with various improvements that were…

AlphaLoRA: Assigning LoRA Experts Based on Layer Training Quality

2024-10-14 · Peijun Qing, Chongyang Gao, Yefan Zhou, Xingjian Diao 외

Parameter-efficient fine-tuning methods, such as Low-Rank Adaptation (LoRA), are known to enhance training efficiency in Large Language Models (LLMs). Due to the limited parameters of LoRA, recent studies seek to combine…

Mixture-of-Experts parameter-efficient fine-tuning

GeoRA: Geometry-Aware Low-Rank Adaptation for RLVR

2026-01-14 · Jiaying Zhang, Lei Shi, Jiguo Li, Jun Xu 외 arxiv

Reinforcement Learning with Verifiable Rewards (RLVR) is a key paradigm for improving large-scale reasoning models. Unlike supervised fine-tuning (SFT), RLVR exhibits distinct optimization dynamics and is sensitive to th…

Reinforcement Learning

Sparsity-Aware Low-Rank Representation for Efficient Fine-Tuning of Large Language Models

2026-01-08 · Longteng Zhang, Sen Wu, Shuai Hou, Zhengyu Qing 외 arxiv

Adapting large pre-trained language models to downstream tasks often entails fine-tuning millions of parameters or deploying costly dense weight updates, which hinders their use in resource-constrained environments. Low-…

Model Compression

HALOC: Hardware-Aware Automatic Low-Rank Compression for Compact Neural Networks

2023-01-20 · Jinqi Xiao, Chengming Zhang, Yu Gong, Miao Yin 외

Low-rank compression is an important model compression strategy for obtaining compact neural network models. In general, because the rank values directly determine the model complexity and model accuracy, proper selectio…

GPU Low-rank compression Model Compression

One Proxy Device Is Enough for Hardware-Aware Neural Architecture Search

2021-11-01 · Bingqian Lu, Jianyi Yang, Weiwen Jiang, Yiyu Shi 외

Convolutional neural networks (CNNs) are used in numerous real-world applications such as vision-based autonomous driving and video content analysis. To run CNN inference on various target devices, hardware-aware neural …

Hardware Aware Neural Architecture Search Neural Architecture Search