Toward General Scene Graph: Integration of Visual Semantic Knowledge with Entity Synset Alignment
Scene graph is a graph representation that explicitly represents high-level semantic knowledge of an image such as objects, attributes of objects and relationships between objects. Various tasks have been proposed for the scene graph, but the problem is that they have a limited vocabulary and biased information due to their own hypothesis. Therefore, results of each task are not generalizable and difficult to be applied to other down-stream tasks. In this paper, we propose Entity Synset Alignment(ESA), which is a method to create a general scene graph by aligning various semantic knowledge efficiently to solve this bias problem. The ESA uses a large-scale lexical database, WordNet and Intersection of Union (IoU) to align the object labels in multiple scene graphs/semantic knowledge. In experiment, the integrated scene graph is applied to the image-caption retrieval task as a down-stream task. We confirm that integrating multiple scene graphs helps to get better representations of images.
Code (0)
등록된 구현이 없습니다.
Tasks
RetrievalSimilar Papers 제목 키워드 기반
Unbiased Video Scene Graph Generation via Visual and Semantic Dual Debiasing
Video Scene Graph Generation (VidSGG) aims to capture dynamic relationships among entities by sequentially analyzing video frames and integrating visual and semantic information. However, VidSGG is challenged by signific…
Graph GenerationScene Graph GenerationTripletVideo scene graph generationGenerating Actionable Robot Knowledge Bases by Combining 3D Scene Graphs with Robot Ontologies
In robotics, the effective integration of environmental data into actionable knowledge remains a significant challenge due to the variety and incompatibility of data formats commonly used in scene descriptions, such as M…
SceneGATE: Scene-Graph based co-Attention networks for TExt visual question answering
Most TextVQA approaches focus on the integration of objects, scene texts and question words by a simple transformer encoder. But this fails to capture the semantic relations between different modalities. The paper propos…
Optical Character RecognitionOptical Character Recognition (OCR)Question AnsweringRelation+3Multimodal Contextualized Semantic Parsing from Speech
We introduce Semantic Parsing in Contextual Environments (SPICE), a task designed to enhance artificial agents' contextual awareness by integrating multimodal inputs with prior contexts. SPICE goes beyond traditional sem…
Data Integrationgraph constructionSemantic ParsingVision-Language Integration for Zero-Shot Scene Understanding in Real-World Environments
Zero-shot scene understanding in real-world settings presents major challenges due to the complexity and variability of natural scenes, where models must recognize new objects, actions, and contexts without prior labeled…
Scene UnderstandingActivity DetectionObject Recognition