Personalized Large Vision-Language Models
The personalization model has gained significant attention in image generation yet remains underexplored for large vision-language models (LVLMs). Beyond generic ones, with personalization, LVLMs handle interactive dialogues using referential concepts (e.g., `Mike and Susan are talking.'') instead of the generic form (e.g., `a boy and a girl are talking.''), making the conversation more customizable and referentially friendly. In addition, PLVM is equipped to continuously add new concepts during a dialogue without incurring additional costs, which significantly enhances the practicality. PLVM proposes Aligner, a pre-trained visual encoder to align referential concepts with the queried images. During the dialogues, it extracts features of reference images with these corresponding concepts and recognizes them in the queried image, enabling personalization. We note that the computational cost and parameter count of the Aligner are negligible within the entire framework. With comprehensive qualitative and quantitative analyses, we reveal the effectiveness and superiority of PLVM.
Code (0)
등록된 구현이 없습니다.
Tasks
Image GenerationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
"This is my unicorn, Fluffy": Personalizing frozen vision-language representations
Large Vision & Language models pretrained on web-scale data provide representations that are invaluable for numerous V&L problems. However, it is unclear how they can be used for reasoning about user-specific visual conc…
Image RetrievalRetrievalSemantic SegmentationSentence+3Improving Personalized Search with Regularized Low-Rank Parameter Updates
Personalized vision-language retrieval seeks to recognize new concepts (e.g. "my dog Fido") from only a few examples. This task is challenging because it requires not only learning a new concept from a few images, but al…
General KnowledgeImage RetrievalNatural Language QueriesRetrievalCommunication-Efficient Personalized Adaptation via Federated-Local Model Merging
Parameter-efficient fine-tuning methods, such as LoRA, offer a practical way to adapt large vision and language models to client tasks. However, this becomes particularly challenging under task-level heterogeneity in fed…
parameter-efficient fine-tuningGeneral KnowledgeConCon-Chi: Concept-Context Chimera Benchmark for Personalized Vision-Language Tasks
While recent Vision-Language (VL) models excel at open-vocabulary tasks it is unclear how to use them with specific or uncommon concepts. Personalized Text-to-Image Retrieval (TIR) or Generation (TIG) are recently in…
Image RetrievalPerPilot: Personalizing VLM-based Mobile Agents via Memory and Exploration
Vision language model (VLM)-based mobile agents show great potential for assisting users in performing instruction-driven tasks. However, these agents typically struggle with personalized instructions -- those containing…