A Zero-shot Learning Method Based on Large Language Models for Multi-modal Knowledge Graph Embedding
Zero-shot learning (ZL) is crucial for tasks involving unseen categories, such as natural language processing, image classification, and cross-lingual transfer. Current applications often fail to accurately infer and handle new relations or entities involving unseen categories, severely limiting their scalability and practicality in open-domain scenarios. ZL learning faces the challenge of effectively transferring semantic information of unseen categories in multi-modal knowledge graph (MMKG) embedding representation learning. In this paper, we propose ZSLLM, a framework for zero-shot embedding learning of MMKGs using large language models (LLMs). We leverage textual modality information of unseen categories as prompts to fully utilize the reasoning capabilities of LLMs, enabling semantic information transfer across different modalities for unseen categories. Through model-based learning, the embedding representation of unseen categories in MMKG is enhanced. Extensive experiments conducted on multiple real-world datasets demonstrate the superiority of our approach compared to state-of-the-art methods.
Code (0)
등록된 구현이 없습니다.
Tasks
Cross-Lingual TransferGraph Embeddingimage-classificationImage ClassificationKnowledge Graph EmbeddingMulti-modal Knowledge GraphRepresentation LearningZero-Shot LearningSimilar Papers 제목 키워드 기반
Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languages
Recently there has been a significant surge in multimodal learning in terms of both image-to-text and text-to-image generation. However, the success is typically limited to English, leaving other languages largely behind…
Image GenerationImage to textLanguage ModelingLanguage Modelling+3Multimodal Knowledge Alignment with Reinforcement Learning
Large language models readily adapt to novel settings, even without task-specific training data. Can their zero-shot capacity be extended to multimodal inputs? In this work, we propose ESPER which extends language-only z…
Audio captioningLanguage ModelingLanguage Modellingreinforcement-learning+3MoqaGPT : Zero-Shot Multi-modal Open-domain Question Answering with Large Language Model
Multi-modal open-domain question answering typically requires evidence retrieval from databases across diverse modalities, such as images, tables, passages, etc. Even Large Language Models (LLMs) like GPT-4 fall short in…
Language ModelingLanguage ModellingLarge Language ModelOpen-Domain Question Answering+2Eyeballing Combinatorial Problems: A Case Study of Using Multimodal Large Language Models to Solve Traveling Salesman Problems
Multimodal Large Language Models (MLLMs) have demonstrated proficiency in processing di-verse modalities, including text, images, and audio. These models leverage extensive pre-existing knowledge, enabling them to addres…
In-Context LearningTraveling Salesman ProblemVisual ReasoningEnabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation
The recent large-scale vision-language pre-training (VLP) of dual-stream architectures (e.g., CLIP) with a tremendous amount of image-text pair data, has shown its superiority on various multimodal alignment tasks. Despi…
Image CaptioningKnowledge DistillationLanguage ModelingLanguage Modelling+4