Prototype-based HyperAdapter for Sample-Efficient Multi-task Tuning

2023-10-18 · Hao Zhao, Jie Fu, Zhaofeng He

Parameter-efficient fine-tuning (PEFT) has shown its effectiveness in adapting the pre-trained language models to downstream tasks while only updating a small number of parameters. Despite the success, most existing methods independently adapt to each task without considering knowledge transfer between tasks and are limited to low-data regimes. To overcome this issue, we propose Prototype-based HyperAdapter (PHA), a novel framework built on the adapter-tuning and hypernetwork. It introduces an instance-dense retriever and a prototypical hypernetwork to generate the conditional modules in a sample-efficient manner. This leads to comparable performance improvements against existing PEFT methods on multi-task learning and few-shot transfer learning. More importantly, when the available data size gets smaller, our method outperforms other strong baselines by a large margin. Based on our extensive empirical experiments across various datasets, we demonstrate that PHA strikes a better trade-off between trainable parameters, accuracy on stream tasks, and sample efficiency.

📄 PDF Abstract BibTeX arXiv:2310.11670

Code (1)

bumble666/pha 공식 구현 pytorch

Tasks

Multi-Task Learning parameter-efficient fine-tuning Transfer Learning

Methods 이 논문이 사용한 방법론

HyperNetwork A HyperNetwork is a network that generates weights for a main network. The behavior of the main network is the same with any usual neural network: it learns to map some raw…

Structured Hyperedge Adaptation for Parameter-Efficient Fine-Tuning of Vision Transformers

2026-06-21 · Edwin Kwadwo Tenagyei, Lei Wang, Ugochukwu Ejike Akpudo, Jun Zhou 외 arxiv

Parameter-efficient fine-tuning (PEFT) has become a practical solution for adapting large pretrained vision transformers (ViTs) to downstream tasks while updating only a small subset of parameters. However, existing adap…

parameter-efficient fine-tuning

Unsupervised Prototype Adapter for Vision-Language Models

2023-08-22 · Yi Zhang, Ce Zhang, Xueting Hu, Zhihai He

Recently, large-scale pre-trained vision-language models (e.g. CLIP and ALIGN) have demonstrated remarkable effectiveness in acquiring transferable visual representations. To leverage the valuable knowledge encoded withi…

Domain Generalization

DroneFINE: Domain-Aware Parameter-Efficient Fine-Tuning of Vision-Language Detectors for Drone Images

2026-07-01 · Ke Wu, Yanan Zhang, Yingjie Gao, Wenhao Li 외 arxiv

Object detection for Unmanned Aerial Vehicles (UAVs) working in open and dynamic environments is a highly challenging task. While Vision-Language Models (VLMs) have offered a powerful solution for universal object detect…

parameter-efficient fine-tuning Object Detection

Multi-Cache Enhanced Prototype Learning for Test-Time Generalization of Vision-Language Models

2025-08-02 · Xinyu Chen, Haotian Zhai, Can Zhang, Xiupeng Shi 외 arxiv

In zero-shot setting, test-time adaptation adjusts pre-trained models using unlabeled data from the test phase to enhance performance on unknown test distributions. Existing cache-enhanced TTA methods rely on a low-entro…

Test-time Adaptation

Unlocking Prototype Potential: An Efficient Tuning Framework for Few-Shot Class-Incremental Learning

2026-02-05 · Shengqin Jiang, Xiaoran Feng, Yuankai Qi, Haokui Zhang 외 arxiv

Few-shot class-incremental learning (FSCIL) seeks to continuously learn new classes from very limited samples while preserving previously acquired knowledge. Traditional methods often utilize a frozen pre-trained feature…

Few-Shot Class-Incremental Learning