FREE-Switch: Frequency-based Dynamic LoRA Switch for Style Transfer
With the growing availability of open-sourced adapters trained on the same diffusion backbone for diverse scenes and objects, combining these pretrained weights enables low-cost customized generation. However, most existing model merging methods are designed for classification or text generation, and when applied to image generation, they suffer from content drift due to error accumulation across multiple diffusion steps. For image-oriented methods, training-based approaches are computationally expensive and unsuitable for edge deployment, while training-free ones use uniform fusion strategies that ignore inter-adapter differences, leading to detail degradation. We find that since different adapters are specialized for generating different types of content, the contribution of each diffusion step carries different significance for each adapter. Accordingly, we propose a frequency-domain importance-driven dynamic LoRA switch method. Furthermore, we observe that maintaining semantic consistency across adapters effectively mitigates detail loss; thus, we design an automatic Generation Alignment mechanism to align generation intents at the semantic level. Experiments demonstrate that our FREE-Switch (Frequency-based Efficient and Dynamic LoRA Switch) framework efficiently combines adapters for different objects and styles, substantially reducing the training cost of high-quality customized generation.
Code (0)
등록된 구현이 없습니다.
Tasks
Image GenerationText GenerationStyle TransferSimilar Papers 제목 키워드 기반
SwitchLoRA: Switched Low-Rank Adaptation Can Learn Full-Rank Information
In the training of large language models, parameter-efficient techniques such as LoRA optimize memory usage and reduce communication overhead and memory usage during the fine-tuning phase. However, applying such techniqu…
Language ModellingLarge Language Modelparameter-efficient fine-tuningLoRA-Switch: Boosting the Efficiency of Dynamic LLM Adapters via System-Algorithm Co-design
Recent literature has found that an effective method to customize or further improve large language models (LLMs) is to add dynamic adapters, such as low-rank adapters (LoRA) with Mixture-of-Experts (MoE) structures. Tho…
Mixture-of-ExpertsLearning Invariant Stabilizing Controllers for Frequency Regulation under Variable Inertia
Declines in cost and concerns about the environmental impact of traditional generation have boosted the penetration of renewables and non-conventional distributed energy resources into the power grid. The intermittent av…
Sample-Efficient Reinforcement Learning with loglog(T) Switching Cost
We study the problem of reinforcement learning (RL) with low (policy) switching cost - a problem well-motivated by real-life RL applications in which deployments of new policies are costly and the number of policy update…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Provably Efficient Q-Learning with Low Switching Cost
We take initial steps in studying PAC-MDP algorithms with limited adaptivity, that is, algorithms that change its exploration policy as infrequently as possible during regret minimization. This is motivated by the diffic…
Q-Learning