MONKEY: Masking ON KEY-Value Activation Adapter for Personalization
Personalizing diffusion models allows users to generate new images that incorporate a given subject, allowing more control than a text prompt. These models often suffer somewhat when they end up just recreating the subject image and ignoring the text prompt. We observe that one popular method for personalization, IP-Adapter, automatically generates masks that segment the subject from the background during inference. We propose to use this automatically generated mask on a second pass to mask the image tokens, thus restricting them to the subject, not the background, allowing the text prompt to attend to the rest of the image. For text prompts describing locations and places, this produces images that accurately depict the subject while definitively matching the prompt. We compare our method to a few other test time personalization methods, and find our method displays high prompt and source image alignment. We also perform a user study to validate whether end users would appreciate our method. Code available at https://github.com/jamesBaker361/monkey
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Monkey Jump : MoE-Style PEFT for Efficient Multi-Task Learning
Mixture-of-experts variants of parameter-efficient fine-tuning enable per-token specialization, but they introduce additional trainable routers and expert parameters, increasing memory usage and training cost. This under…
parameter-efficient fine-tuningMulti-Task LearningEfficient Multi-Adapter LLM Serving via Cross-Model KV-Cache Reuse with Activated LoRA
Modern large language model (LLM) systems increasingly rely on multi-turn pipelines that are composed of multiple task-specific adapters, yet existing serving frameworks remain inefficient, incurring substantial recomput…
Adaptive Rank Allocation: Speeding Up Modern Transformers with RaNA Adapters
Large Language Models (LLMs) are computationally intensive, particularly during inference. Neuron-adaptive techniques, which selectively activate neurons in Multi-Layer Perceptron (MLP) layers, offer some speedups but su…
Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models
Large Multimodal Models (LMMs) have shown promise in vision-language tasks but struggle with high-resolution input and detailed scene understanding. Addressing these challenges, we introduce Monkey to enhance LMM capabil…
Image CaptioningMMR totalQuestion AnsweringScene Understanding+2Gate-and-Merge: Zero-shot Compositional Personalization of Vision Language Models
This paper tackles compositional personalization of vision-language models (VLMs). In this problem, multiple user-defined concepts must be recognized or described jointly at test time. We introduce Gate-and-Merge, a zero…