GRAPHGPT-O: Synergistic Multimodal Comprehension and Generation on Graphs
The rapid development of Multimodal Large Language Models (MLLMs) has enabled the integration of multiple modalities, including texts and images, within the large language model (LLM) framework. However, texts and images are usually interconnected, forming a multimodal attributed graph (MMAG). It is underexplored how MLLMs can incorporate the relational information (\textit{i.e.}, graph structure) and semantic information (\textit{i.e.,} texts and images) on such graphs for multimodal comprehension and generation. In this paper, we propose GraphGPT-o, which supports omni-multimodal understanding and creation on MMAGs. We first comprehensively study linearization variants to transform semantic and structural information as input for MLLMs. Then, we propose a hierarchical aligner that enables deep graph encoding, bridging the gap between MMAGs and MLLMs. Finally, we explore the inference choices, adapting MLLM to interleaved text and image generation in graph scenarios. Extensive experiments on three datasets from different domains demonstrate the effectiveness of our proposed method. Datasets and codes will be open-sourced upon acceptance.
Code (0)
등록된 구현이 없습니다.
Tasks
Image GenerationLanguage ModelingLanguage ModellingLarge Language ModelSimilar Papers 제목 키워드 기반
A Graph is Worth $K$ Words: Euclideanizing Graph using Pure Transformer
Can we model Non-Euclidean graphs as pure language or even Euclidean vectors while retaining their inherent information? The Non-Euclidean property have posed a long term challenge in graph modeling. Despite recent graph…
DecoderGraph ClassificationGraph GenerationGraph Representation Learning+1MR-MLLM: Mutual Reinforcement of Multimodal Comprehension and Vision Perception
In recent years, multimodal large language models (MLLMs) have shown remarkable capabilities in tasks like visual question answering and common sense reasoning, while visual perception models have made significant stride…
Common Sense ReasoningLanguage ModellingLarge Language ModelMultimodal Large Language Model+4DreamLLM: Synergistic Multimodal Comprehension and Creation
This paper presents DreamLLM, a learning framework that first achieves versatile Multimodal Large Language Models (MLLMs) empowered with frequently overlooked synergy between multimodal comprehension and creation. DreamL…
multimodal generationVisual Question AnsweringZero-Shot LearningZero-Shot Text-to-Image GenerationGraphGPT: Graph Learning with Generative Pre-trained Transformers
We introduce \textit{GraphGPT}, a novel model for Graph learning by self-supervised Generative Pre-training Transformers. Our model transforms each graph or sampled subgraph into a sequence of tokens representing the nod…
DecoderGraph Learningscientific discoveryGraphGPT: Graph Instruction Tuning for Large Language Models
Graph Neural Networks (GNNs) have evolved to understand graph structures through recursive exchanges and aggregations among nodes. To enhance robustness, self-supervised learning (SSL) has become a vital tool for data au…
Data AugmentationGraph LearningSelf-Supervised LearningZero-Shot Learning