Alpha-CLIP: A CLIP Model Focusing on Wherever You Want
Contrastive Language-Image Pre-training (CLIP) plays an essential role in extracting valuable content information from images across diverse tasks. It aligns textual and visual modalities to comprehend the entire image, including all the details, even those irrelevant to specific tasks. However, for a finer understanding and controlled editing of images, it becomes crucial to focus on specific regions of interest, which can be indicated as points, masks, or boxes by humans or perception models. To fulfill the requirements, we introduce Alpha-CLIP, an enhanced version of CLIP with an auxiliary alpha channel to suggest attentive regions and fine-tuned with constructed millions of RGBA region-text pairs. Alpha-CLIP not only preserves the visual recognition ability of CLIP but also enables precise control over the emphasis of image contents. It demonstrates effectiveness in various tasks, including but not limited to open-world recognition, multimodal large language models, and conditional 2D / 3D generation. It has a strong potential to serve as a versatile tool for image-related tasks.
Code (1)
Tasks
3D GenerationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Taming Fat-Tailed ("Heavier-Tailed'' with Potentially Infinite Variance) Noise in Federated Learning
A key assumption in most existing works on FL algorithms' convergence analysis is that the noise in stochastic first-order information has a finite variance. Although this assumption covers all light-tailed (i.e., sub-ex…
Federated LearningThey're All Doctors: Synthesizing Diverse Counterfactuals to Mitigate Associative Bias
Vision Language Models (VLMs) such as CLIP are powerful models; however they can exhibit unwanted biases, making them less safe when deployed directly in applications such as text-to-image, text-to-video retrievals, reve…
AllcounterfactualFairnessImage RetrievalFontCLIP: A Semantic Typography Visual-Language Model for Multilingual Font Applications
Acquiring the desired font for various design tasks can be challenging and requires professional typographic knowledge. While previous font retrieval or generation works have alleviated some of these difficulties, they o…
AttributeDescriptiveLanguage ModelingLanguage Modelling+1Frame-Difference Guided Dynamic Region Perception for CLIP Adaptation in Text-Video Retrieval
With the rapid growth of video data, text-video retrieval technology has become increasingly important in numerous application scenarios such as recommendation and search. Early text-video retrieval methods suffer from t…
Video AlignmentVideo RetrievalHigh Probability Analysis for Non-Convex Stochastic Optimization with Clipping
Gradient clipping is a commonly used technique to stabilize the training process of neural networks. A growing body of studies has shown that gradient clipping is a promising technique for dealing with the heavy-tailed b…
Stochastic Optimization