VisualSem: A High-quality Knowledge Graph for Vision and Language
An exciting frontier in natural language understanding (NLU) and generation (NLG) calls for (vision-and-) language models that can efficiently access external structured knowledge repositories. However, many existing knowledge bases only cover limited domains, or suffer from noisy data, and most of all are typically hard to integrate into neural language pipelines. To fill this gap, we release VisualSem: a high-quality knowledge graph (KG) which includes nodes with multilingual glosses, multiple illustrative images, and visually relevant relations. We also release a neural multi-modal retrieval model that can use images or sentences as inputs and retrieves entities in the KG. This multi-modal retrieval model can be integrated into any (neural network) model pipeline. We encourage the research community to use VisualSem for data augmentation and/or as a source of grounding, among other possible uses. VisualSem as well as the multi-modal retrieval models are publicly available and can be downloaded in this URL: https://github.com/iacercalixto/visualsem
Code (1)
Tasks
Data AugmentationNatural Language UnderstandingRetrievalVocal Bursts Intensity PredictionSimilar Papers 제목 키워드 기반
Endowing Language Models with Multimodal Knowledge Graph Representations
We propose a method to make natural language understanding models more parameter efficient by storing knowledge in an external knowledge graph (KG) and retrieving from this KG using a dense index. Given (possibly multili…
Multilingual Named Entity Recognitionnamed-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)+1PathISE: Learning Informative Path Supervision for Knowledge Graph Question Answering
Knowledge Graph Question Answering (KGQA) aims to answer user questions by reasoning over Knowledge Graphs (KGs). Recent KGQA methods mainly follow the retrieval-augmented generation paradigm to ground Large Language Mod…
Graph Question AnsweringKnowledge GraphsKnowledge Graph Curation: A Practical Framework
Knowledge Graphs (KGs) have shown to be very important for applications such as personal assistants, question-answering systems, and search engines. Therefore, it is crucial to ensure their high quality. However, KGs ine…
Knowledge GraphsMissing ValuesQuestion AnsweringKnowledge Graph Contrastive Learning for Recommendation
Knowledge Graphs (KGs) have been utilized as useful side information to improve recommendation quality. In those recommender systems, knowledge graph information often contains fruitful facts and inherent semantic relate…
Contrastive LearningGeneral KnowledgeKnowledge GraphsRecommendation Systems+1Transductive Zero-Shot Learning with Adaptive Structural Embedding
Zero-shot learning (ZSL) endows the computer vision system with the inferential capability to recognize instances of a new category that has never seen before. Two fundamental challenges in it are visual-semantic embeddi…
Domain AdaptationZero-Shot Learning