Knowledge Transfer Across Modalities with Natural Language Supervision
We present a way to learn novel concepts by only using their textual description. We call this method Knowledge Transfer. Similarly to human perception, we leverage cross-modal interaction to introduce new concepts. We hypothesize that in a pre-trained visual encoder there are enough low-level features already learned (e.g. shape, appearance, color) that can be used to describe previously unknown high-level concepts. Provided with a textual description of the novel concept, our method works by aligning the known low-level features of the visual encoder to its high-level textual description. We show that Knowledge Transfer can successfully introduce novel concepts in multimodal models, in a very efficient manner, by only requiring a single description of the target concept. Our approach is compatible with both separate textual and visual encoders (e.g. CLIP) and shared parameters across modalities. We also show that, following the same principle, Knowledge Transfer can improve concepts already known by the model. Leveraging Knowledge Transfer we improve zero-shot performance across different tasks such as classification, segmentation, image-text retrieval, and captioning.
Code (0)
등록된 구현이 없습니다.
Tasks
Image-text RetrievalNovel ConceptsText RetrievalTransfer LearningSimilar Papers 제목 키워드 기반
CLIP-RL: Aligning Language and Policy Representations for Task Transfer in Reinforcement Learning
Recently, there has been an increasing need to develop agents capable of solving multiple tasks within the same environment, especially when these tasks are naturally associated with language. In this work, we propose a …
Reinforcement LearningCan Offline Reinforcement Learning Help Natural Language Understanding?
Pre-training has been a useful method for learning implicit transferable knowledge and it shows the benefit of offering complementary features across different modalities. Recent work mainly focuses on the modalities suc…
Language ModelingLanguage ModellingNatural Language UnderstandingOffline RL+3Distilling Cross-Modal Knowledge via Feature Disentanglement
Knowledge distillation (KD) has proven highly effective for compressing large models and enhancing the performance of smaller ones. However, its effectiveness diminishes in cross-modal scenarios, such as vision-to-langua…
Knowledge DistillationEverything is a Video: Unifying Modalities through Next-Frame Prediction
Multimodal learning, which involves integrating information from various modalities such as text, images, audio, and video, is pivotal for numerous complex tasks like visual question answering, cross-modal retrieval, and…
Caption GenerationCross-Modal RetrievalImage to textQuestion Answering+2A Zero-shot Learning Method Based on Large Language Models for Multi-modal Knowledge Graph Embedding
Zero-shot learning (ZL) is crucial for tasks involving unseen categories, such as natural language processing, image classification, and cross-lingual transfer. Current applications often fail to accurately infer and han…
Cross-Lingual TransferGraph Embeddingimage-classificationImage Classification+4