Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Survey
Knowledge Graphs (KGs) play a pivotal role in advancing various AI applications, with the semantic web community's exploration into multi-modal dimensions unlocking new avenues for innovation. In this survey, we carefully review over 300 articles, focusing on KG-aware research in two principal aspects: KG-driven Multi-Modal (KG4MM) learning, where KGs support multi-modal tasks, and Multi-Modal Knowledge Graph (MM4KG), which extends KG studies into the MMKG realm. We begin by defining KGs and MMKGs, then explore their construction progress. Our review includes two primary task categories: KG-aware multi-modal learning tasks, such as Image Classification and Visual Question Answering, and intrinsic MMKG tasks like Multi-modal Knowledge Graph Completion and Entity Alignment, highlighting specific research trajectories. For most of these tasks, we provide definitions, evaluation benchmarks, and additionally outline essential insights for conducting relevant research. Finally, we discuss current challenges and identify emerging trends, such as progress in Large Language Modeling and Multi-modal Pre-training strategies. This survey aims to serve as a comprehensive reference for researchers already involved in or considering delving into KG and multi-modal learning research, offering insights into the evolving landscape of MMKG research and supporting future work.
Code (6)
Tasks
ArticlesEntity Alignmentimage-classificationImage ClassificationKnowledge Graph CompletionKnowledge GraphsLanguage ModelingLanguage ModellingMulti-modal Knowledge GraphSurveyVisual Question AnsweringSimilar Papers 제목 키워드 기반
VL-KGE: Vision-Language Models Meet Knowledge Graph Embeddings
Real-world multimodal knowledge graphs (MKGs) are inherently heterogeneous, modeling entities that are associated with diverse modalities. Traditional knowledge graph embedding (KGE) methods excel at learning continuous …
Knowledge Graph EmbeddingKnowledge GraphsLink PredictionWhen Graph meets Multimodal: Benchmarking on Multimodal Attributed Graphs Learning
Multimodal attributed graphs (MAGs) are prevalent in various real-world scenarios and generally contain two kinds of knowledge: (a) Attribute knowledge is mainly supported by the attributes of different modalities contai…
AttributeBenchmarkingRepresentation LearningMMPKUBase: A Comprehensive and High-quality Chinese Multi-modal Knowledge Graph
Multi-modal knowledge graphs have emerged as a powerful approach for information representation, combining data from different modalities such as text, images, and videos. While several such graphs have been constructed …
AttributeContrastive LearningKnowledge GraphsMulti-modal Knowledge Graph+3Multimodal Reasoning with Multimodal Knowledge Graph
Multimodal reasoning with large language models (LLMs) often suffers from hallucinations and the presence of deficient or outdated knowledge within LLMs. Some approaches have sought to mitigate these issues by employing …
cross-modal alignmentGraph AttentionKnowledge GraphsMultimodal Reasoning+1Hierarchical Network Fusion for Multi-Modal Electron Micrograph Representation Learning with Foundational Large Language Models
Characterizing materials with electron micrographs is a crucial task in fields such as semiconductors and quantum materials. The complex hierarchical structure of micrographs often poses challenges for traditional classi…
Representation Learning