CK-Transformer: Commonsense Knowledge Enhanced Transformers for Referring Expression Comprehension
The task of multimodal referring expression comprehension (REC), aiming at localizing an image region described by a natural language expression, has recently received increasing attention within the research comminity. In this paper, we specifically focus on referring expression comprehension with commonsense knowledge (KB-Ref), a task which typically requires reasoning beyond spatial, visual or semantic information. We propose a novel framework for Commonsense Knowledge Enhanced Transformers (CK-Transformer) which effectively integrates commonsense knowledge into the representations of objects in an image, facilitating identification of the target objects referred to by the expressions. We conduct extensive experiments on several benchmarks for the task of KB-Ref. Our results show that the proposed CK-Transformer achieves a new state of the art, with an absolute improvement of 3.14% accuracy over the existing state of the art.
Code (1)
Tasks
Referring ExpressionReferring Expression ComprehensionSimilar Papers 제목 키워드 기반
SGEITL: Scene Graph Enhanced Image-Text Learning for Visual Commonsense Reasoning
Answering complex questions about images is an ambitious goal for machine intelligence, which requires a joint understanding of images, text, and commonsense knowledge, as well as a strong reasoning ability. Recently, mu…
Visual Commonsense ReasoningCOMET: Commonsense Transformers for Automatic Knowledge Graph Construction
We present the first comprehensive study on automatic knowledge base construction for two prevalent commonsense knowledge graphs: ATOMIC (Sap et al., 2019) and ConceptNet (Speer et al., 2017). Contrary to many convention…
graph constructionKnowledge Base ConstructionKnowledge GraphsKARNA at COIN Shared Task 1: Bidirectional Encoder Representations from Transformers with relational knowledge for machine comprehension with common sense
This paper describes our model for COmmonsense INference in Natural Language Processing (COIN) shared task 1: Commonsense Inference in Everyday Narrations. This paper explores the use of Bidirectional Encoder Representat…
Common Sense ReasoningReading ComprehensionLearning from Missing Relations: Contrastive Learning with Commonsense Knowledge Graphs for Commonsense Inference
Commonsense inference poses a unique challenge to reason and generate the physical, social, and causal conditions of a given event. Existing approaches to commonsense inference utilize commonsense transformers, which are…
Contrastive LearningDiversityKnowledge GraphsLearning from Missing Relations: Contrastive Learning with Commonsense Knowledge Graphs for Commonsense Inference
Commonsense inference poses a unique challenge to reason and generate the physical, social, and causal conditions of a given event. Existing approaches to commonsense inference utilize commonsense transformers, which are…
Contrastive LearningDiversityKnowledge Graphs