MoSE: Modality Split and Ensemble for Multimodal Knowledge Graph Completion
Multimodal knowledge graph completion (MKGC) aims to predict missing entities in MKGs. Previous works usually share relation representation across modalities. This results in mutual interference between modalities during training, since for a pair of entities, the relation from one modality probably contradicts that from another modality. Furthermore, making a unified prediction based on the shared relation representation treats the input in different modalities equally, while their importance to the MKGC task should be different. In this paper, we propose MoSE, a Modality Split representation learning and Ensemble inference framework for MKGC. Specifically, in the training phase, we learn modality-split relation embeddings for each modality instead of a single modality-shared one, which alleviates the modality interference. Based on these embeddings, in the inference phase, we first make modality-split predictions and then exploit various ensemble methods to combine the predictions with different weights, which models the modality importance dynamically. Experimental results on three KG datasets show that MoSE outperforms state-of-the-art MKGC methods. Codes are available at https://github.com/OreOZhao/MoSE4MKGC.
Code (1)
Tasks
Knowledge Graph CompletionRelationRepresentation LearningSimilar Papers 제목 키워드 기반
Graph-based Fine-grained Multimodal Attention Mechanism for Sentiment Analysis
Multimodal sentiment analysis is a popular research area in natural language processing. Mainstream multimodal learning models barely consider that the visual and acoustic behaviors often have a much higher temporal freq…
multimodal interactionMultimodal Sentiment AnalysisSentiment AnalysisBenchmarking and Bridging Emotion Conflicts for Multimodal Emotion Reasoning
Despite their strong performance in multimodal emotion reasoning, existing Multimodal Large Language Models (MLLMs) often overlook the scenarios involving emotion conflicts, where emotional cues from different modalities…
VideoAdviser: Video Knowledge Distillation for Multimodal Transfer Learning
Multimodal transfer learning aims to transform pretrained representations of diverse modalities into a common domain space for effective multimodal fusion. However, conventional systems are typically built on the assumpt…
Knowledge DistillationregressionSentiment AnalysisTransfer LearningCorrect When Paired, Wrong When Split: Decoupling and Editing Modality-Specific Neurons in MLLMs
Although Knowledge Editing provides an efficient mechanism for updating the knowledge of Multimodal Large Language Models (MLLMs), we find that current paradigms still suffer from an important yet remain underexplored is…
knowledge editingTri-Subspaces Disentanglement for Multimodal Sentiment Analysis
Multimodal Sentiment Analysis (MSA) integrates language, visual, and acoustic modalities to infer human sentiment. Most existing methods either focus on globally shared representations or modality-specific features, whil…
Multimodal Intent RecognitionMultimodal Sentiment Analysis