홈 › Papers

RDP LoRA: Geometry-Driven Identification for Parameter-Efficient Adaptation in Large Language Models

2026-04-21 · Yusuf Çelebi, Yağız Asker, Özay Ezerceli, Mahmoud ElHussieni, Selva Taş, Reyhan Bayraktar, Fatma Betül Terzioğlu arxiv

Fine-tuning Large Language Models (LLMs) remains structurally uncertain despite parameter-efficient methods such as Low-Rank Adaptation (LoRA), as the layer-specific roles of internal representations are poorly understood, leading to heuristic decisions about where adaptation should be applied. We model the evolution of hidden states as a high-dimensional geometric trajectory and propose using the Ramer-Douglas-Peucker (RDP) algorithm, a parameter-free and training-free polygon simplification method that preserves global structural transitions while eliminating locally redundant changes, to identify critical breakpoints along the representation path. Crucially, we use these geometric pivots not merely for analysis, but as a direct decision signal for determining which layers should be adapted during parameter-efficient fine-tuning. By integrating this geometry-aware layer selection strategy into LoRA fine-tuning of Qwen3-8B-Base, we achieve superior performance on MMLU-Math using only 13 RDP-selected layers (81.67%), significantly outperforming both full 36-layer adaptation (79.32%) and random 13-layer selection (75.56%), as well as the baseline Qwen3-8B-Base model (74.25%). These results demonstrate that leveraging the intrinsic geometry of representation trajectories provides a robust, interpretable, and training-free signal for optimizing layer selection during model adaptation.

📄 PDF Abstract BibTeX arXiv:2604.19321

Code (0)

등록된 구현이 없습니다.

Tasks

parameter-efficient fine-tuning

Geometry-Preserving Orthonormal Initialization for Low-Rank Adaptation in RLVR

2026-06-30 · Ruijia Zhang, Jiacheng Zhu, Hanqing Zhu, Laixi Shi arxiv

Low-rank adaptation (LoRA) and its variants enable parameter-efficient fine-tuning of large language models under the supervised fine-tuning (SFT) paradigm. However, their efficacy and behavior under Reinforcement learni…

parameter-efficient fine-tuning Reinforcement Learning Mathematical Reasoning

CoLoRA: Continuous low-rank adaptation for reduced implicit neural modeling of parameterized partial differential equations

2024-02-22 · Jules Berman, Benjamin Peherstorfer

This work introduces reduced models based on Continuous Low Rank Adaptation (CoLoRA) that pre-train neural networks for a given partial differential equation and then continuously adapt low-rank weights in time to rapidl…

TalkLoRA: Low-Rank Adaptation for Speech-Driven Animation

2024-08-25 · Jack Saunders, Vinay Namboodiri

Speech-driven facial animation is important for many applications including TV, film, video games, telecommunication and AR/VR. Recently, transformers have been shown to be extremely effective for this task. However, we …

Chunking

MetaCURE: Meta Reinforcement Learning with Empowerment-Driven Exploration

2020-06-15 · Jin Zhang, Jianhao Wang, Hao Hu, Tong Chen 외

Meta reinforcement learning (meta-RL) extracts knowledge from previous tasks and achieves fast adaptation to new tasks. Despite recent progress, efficient exploration in meta-RL remains a key challenge in sparse-reward t…

Efficient Exploration Meta Reinforcement Learning MuJoCo reinforcement-learning+2

Orthogonal Low-rank Adaptation in Lie Groups for Continual Learning of Large Language Models

2025-09-07 · Kefan Cao, Shuaicheng Wu arxiv

Large language models (LLMs) suffer from catastrophic forgetting in sequential multi-task learning. Existing parameter regularization methods (e.g., O-LoRA, N-LoRA) mitigate interference via low-rank subspace orthogonali…

Multi-Task Learning Continual Learning