Transferring Knowledge from Vision to Language: How to Achieve it and how to Measure it?
Large language models are known to suffer from the hallucination problem in that they are prone to output statements that are false or inconsistent, indicating a lack of knowledge. A proposed solution to this is to provide the model with additional data modalities that complements the knowledge obtained through text. We investigate the use of visual data to complement the knowledge of large language models by proposing a method for evaluating visual knowledge transfer to text for uni- or multimodal language models. The method is based on two steps, 1) a novel task querying for knowledge of memory colors, i.e. typical colors of well-known objects, and 2) filtering of model training data to clearly separate knowledge contributions. Additionally, we introduce a model architecture that involves a visual imagination step and evaluate it with our proposed method. We find that our method can successfully be used to measure visual knowledge transfer capabilities in models and that our novel model architecture shows promising results for leveraging multimodal knowledge in a unimodal setting.
Code (0)
등록된 구현이 없습니다.
Tasks
HallucinationTransfer LearningSimilar Papers 제목 키워드 기반
Revisiting Classifier: Transferring Vision-Language Models for Video Recognition
Transferring knowledge from task-agnostic pre-trained deep models for downstream tasks is an important topic in computer vision research. Along with the growth of computational capacity, we now have open-source vision-la…
Action ClassificationAction RecognitionClassificationLanguage Modelling+3Similarity-based Knowledge Transfer for Cross-Domain Reinforcement Learning
Transferring knowledge in cross-domain reinforcement learning is a challenging setting in which learning is accelerated by reusing knowledge from a task with different observation and/or action space. However, it is ofte…
MuJoCoreinforcement-learningReinforcement LearningTransfer LearningCLIP-based Synergistic Knowledge Transfer for Text-based Person Retrieval
Text-based Person Retrieval (TPR) aims to retrieve the target person images given a textual query. The primary challenge lies in bridging the substantial gap between vision and language modalities, especially when dealin…
Image to textPerson RetrievalRetrievalText based Person Retrieval+2Transfer Generative Adversarial Networks (T-GAN)-based Terahertz Channel Modeling
Terahertz (THz) communications are envisioned as a promising technology for 6G and beyond wireless systems, providing ultra-broad bandwidth and thus Terabit-per-second (Tbps) data rates. However, as foundation of designi…
Generative Adversarial NetworkSSIMTransfer LearningDAIT: Distillation from Vision-Language Models to Lightweight Classifiers with Adaptive Intermediate Teacher Transfer
Large-scale Vision-Language Models (VLMs) encode rich multimodal semantics that are highly beneficial for fine-grained visual categorization (FGVC). However, their prohibitive computational cost hinders practical deploym…
Knowledge Distillation