Yo'Chameleon: Personalized Vision and Language Generation
Large Multimodal Models (e.g., GPT-4, Gemini, Chameleon) have evolved into powerful tools with millions of users. However, they remain generic models and lack personalized knowledge of specific user concepts. Previous work has explored personalization for text generation, yet it remains unclear how these methods can be adapted to new modalities, such as image generation. In this paper, we introduce Yo'Chameleon, the first attempt to study personalization for large multimodal models. Given 3-5 images of a particular concept, Yo'Chameleon leverages soft-prompt tuning to embed subject-specific information to (i) answer questions about the subject and (ii) recreate pixel-level details to produce images of the subject in new contexts. Yo'Chameleon is trained with (i) a self-prompting optimization mechanism to balance performance across multiple modalities, and (ii) a "soft-positive" image generation approach to enhance image quality in a few-shot setting. Our qualitative and quantitative analyses reveal that Yo'Chameleon can learn concepts more efficiently using fewer tokens and effectively encode visual attributes, outperforming prompting baselines.
Code (0)
등록된 구현이 없습니다.
Tasks
Image GenerationText GenerationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
YoChameleon: Personalized Vision and Language Generation
Large Multimodal Models (e.g., GPT-4, Gemini, Chameleon) have evolved into powerful tools with millions of users. However, they remain generic models and lack personalized knowledge of specific user concepts. Previous wo…
Image GenerationText GenerationCodeChameleon: Personalized Encryption Framework for Jailbreaking Large Language Models
Adversarial misuse, particularly through `jailbreaking' that circumvents a model's safety and ethical protocols, poses a significant challenge for Large Language Models (LLMs). This paper delves into the mechanisms behin…
Code CompletionResponse GenerationPersonalized Privacy Protection Mask Against Unauthorized Facial Recognition
Face recognition (FR) can be abused for privacy intrusion. Governments, private companies, or even individual attackers can collect facial images by web scraping to build an FR system identifying human faces without thei…
DiversityEnsemble LearningFace RecognitionBeyond Fixed Psychological Personas: State Beats Trait, but Language Models are State-Blind
User interactions with language models vary due to static properties of the user (trait) and the specific context of the interaction (state). However, existing persona datasets (like PersonaChat, PANDORA etc.) capture on…
Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models
Large language models (LLMs) have achieved remarkable progress in solving various natural language processing tasks due to emergent reasoning abilities. However, LLMs have inherent limitations as they are incapable of ac…
Logical Reasoning