Device Tuning for Multi-Task Large Model
Unsupervised pre-training approaches have achieved great success in many fields such as Computer Vision (CV), Natural Language Processing (NLP) and so on. However, compared to typical deep learning models, pre-training or even fine-tuning the state-of-the-art self-attention models is extremely expensive, as they require much more computational and memory resources. It severely limits their applications and success in a variety of domains, especially for multi-task learning. To improve the efficiency, we propose Device Tuning for the efficient multi-task model, which is a massively multitask framework across the cloud and device and is designed to encourage learning of representations that generalize better to many different tasks. Specifically, we design Device Tuning architecture of a multi-task model that benefits both cloud modelling and device modelling, which reduces the communication between device and cloud by representation compression. Experimental results demonstrate the effectiveness of our proposed method.
Code (0)
등록된 구현이 없습니다.
Tasks
modelMulti-Task LearningUnsupervised Pre-trainingSimilar Papers 제목 키워드 기반
Enabling Efficient On-Device Fine-Tuning of LLMs Using Only Inference Engines
Large Language Models (LLMs) are currently pre-trained and fine-tuned on large cloud servers. The next frontier is LLM personalization, where a foundation model can be fine-tuned with user/task-specific data. Given the s…
parameter-efficient fine-tuningText GenerationEfficient Deployment of Large Language Models on Resource-constrained Devices
Deploying Large Language Models (LLMs) on resource-constrained (or weak) devices presents significant challenges due to limited resources and heterogeneous data distribution. To address the data concern, it is necessary …
Federated Learningparameter-efficient fine-tuningPrivacy PreservingIncentivizing Multi-Tenant Split Federated Learning for Foundation Models at the Network Edge
Foundation models (FMs) such as GPT-4 exhibit exceptional generative capabilities across diverse downstream tasks through fine-tuning. Split Federated Learning (SFL) facilitates privacy-preserving FM fine-tuning on resou…
Federated LearningPrivacy PreservingPocketLLM: Enabling On-Device Fine-Tuning for Personalized LLMs
Recent advancements in large language models (LLMs) have indeed showcased their impressive capabilities. On mobile devices, the wealth of valuable, non-public data generated daily holds great promise for locally fine-tun…
HD-PiSSA: High-Rank Distributed Orthogonal Adaptation
Existing parameter-efficient fine-tuning (PEFT) methods for large language models (LLMs), such as LoRA and PiSSA, constrain model updates to low-rank subspaces, limiting their expressiveness and leading to suboptimal per…
Code GenerationGPUMulti-Task Learningparameter-efficient fine-tuning