홈 › Papers

AdUE: Improving uncertainty estimation head for LoRA adapters in LLMs

2025-05-21 · Artem Zabolotnyi, Roman Makarov, Mile Mitrovic, Polina Proskura, Oleg Travkin, Roman Alferov, Alexey Zaytsev

Uncertainty estimation remains a critical challenge in adapting pre-trained language models to classification tasks, particularly under parameter-efficient fine-tuning approaches such as adapters. We introduce AdUE1, an efficient post-hoc uncertainty estimation (UE) method, to enhance softmax-based estimates. Our approach (1) uses a differentiable approximation of the maximum function and (2) applies additional regularization through L2-SP, anchoring the fine-tuned head weights and regularizing the model. Evaluations on five NLP classification datasets across four language models (RoBERTa, ELECTRA, LLaMA-2, Qwen) demonstrate that our method consistently outperforms established baselines such as Mahalanobis distance and softmax response. Our approach is lightweight (no base-model changes) and produces better-calibrated confidence.

📄 PDF Abstract BibTeX arXiv:2505.15443

Code (0)

등록된 구현이 없습니다.

Tasks

parameter-efficient fine-tuning

Methods 이 논문이 사용한 방법론

Refunds@Expedia|||How do I get a full refund from Expedia? “How do I get a full refund from Expedia? How do I get a full refund from Expedia? – Call ☎️ +1-(888) 829 (0881) or +1-805-330-4056 or +1-805-330-4056 for Quick Help &…

Attention 설명 없음

Linear Warmup With Linear Decay Linear Warmup With Linear Decay is a learning rate schedule in which we increase the learning rate linearly for $n$ updates and then linearly decay afterwards.

Attention Dropout Attention Dropout is a type of dropout used in attention-based architectures, where elements are randomly dropped out of the…

WordPiece 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Residual Connection 설명 없음

Weight Decay 설명 없음

LoRA ensembles for large language model fine-tuning

2023-09-29 · Xi Wang, Laurence Aitchison, Maja Rudolph

Finetuned LLMs often exhibit poor uncertainty quantification, manifesting as overconfidence, poor calibration, and unreliable prediction results on test data or out-of-distribution samples. One approach commonly used in …

Language Modeling Language Modelling Large Language Model model+2

EulerLoRA: Rank-Driven Jump Dynamics for Calibrated Parameter-Efficient Fine-Tuning

2026-08-02 · Srinivas Anumasa, Dianbo Liu arxiv

Low-Rank Adaptation (LoRA) enables parameter-efficient fine-tuning, but standard LoRA produces a single deterministic model and does not directly support predictive uncertainty estimation. We introduce EulerLoRA, a stoch…

parameter-efficient fine-tuning Out-of-Distribution Detection

Block-Diagonal LoRA for Eliminating Communication Overhead in Tensor Parallel LoRA Serving

2025-10-27 · Xinyu Wang, Jonas M. Kübler, Kailash Budhathoki, Yida Wang 외 arxiv

When serving a single base LLM with several different LoRA adapters simultaneously, the adapters cannot simply be merged with the base model's weights as the adapter swapping would create overhead and requests using diff…

Beyond Feature Fusion: Contextual Bayesian PEFT for Multimodal Uncertainty Estimation

2026-04-17 · Habibeh Naderi, Behrouz Haji Soleimani, Stan Matwin arxiv

We introduce CoCo-LoRA, a multimodal, uncertainty-aware parameter-efficient fine-tuning method for text prediction tasks accompanied by audio context. Existing PEFT approaches such as LoRA are efficient but typically det…