ImageBind: One Embedding Space To Bind Them All
We present ImageBind, an approach to learn a joint embedding across six different modalities - images, text, audio, depth, thermal, and IMU data. We show that all combinations of paired data are not necessary to train such a joint embedding, and only image-paired data is sufficient to bind the modalities together. ImageBind can leverage recent large scale vision-language models, and extends their zero-shot capabilities to new modalities just by using their natural pairing with images. It enables novel emergent applications 'out-of-the-box' including cross-modal retrieval, composing modalities with arithmetic, cross-modal detection and generation. The emergent capabilities improve with the strength of the image encoder and we set a new state-of-the-art on emergent zero-shot recognition tasks across modalities, outperforming specialist supervised models. Finally, we show strong few-shot recognition results outperforming prior work, and that ImageBind serves as a new way to evaluate vision models for visual and non-visual tasks.
Code (3)
Tasks
AllCross-Modal RetrievalMultimodal Deep LearningRetrievalSound Prompted Semantic SegmentationSpeech Prompted Semantic SegmentationTemporal Relation ExtractionZero-shot Audio ClassificationZero-shot Classification (unified classes)Zero-Shot Environment Sound ClassificationZero-Shot LearningZero-shot Scene Classification (unified classes)Zero-shot Text to Audio RetrievalZero-Shot Video RetrievalSimilar Papers 제목 키워드 기반
ImageBind-LLM: Multi-modality Instruction Tuning
We present ImageBind-LLM, a multi-modality instruction tuning method of large language models (LLMs) via ImageBind. Existing works mainly focus on language and image instruction tuning, different from which, our ImageBin…
Instruction FollowingText GenerationPerformance of Gaussian Mixture Model Classifiers on Embedded Feature Spaces
Data embeddings with CLIP and ImageBind provide powerful features for the analysis of multimedia and/or multimodal data. We assess their performance here for classification using a Gaussian Mixture models (GMMs) based la…
Image ClassificationFreeBind: Free Lunch in Unified Multimodal Space via Knowledge Fusion
Unified multi-model representation spaces are the foundation of multimodal understanding and generation. However, the billions of model parameters and catastrophic forgetting problems make it challenging to further enhan…
UniBind: LLM-Augmented Unified and Balanced Representation Space to Bind Them All
We present UniBind, a flexible and efficient approach that learns a unified representation space for seven diverse modalities-- images, text, audio, point cloud, thermal, video, and event data. Existing works, eg., Image…
AllContrastive LearningZero-Shot LearningPandaGPT: One Model To Instruction-Follow Them All
We present PandaGPT, an approach to emPower large lANguage moDels with visual and Auditory instruction-following capabilities. Our pilot experiments show that PandaGPT can perform complex tasks such as detailed image des…
AllImage DescriptionInstruction Following