Zero-Shot Learning with Knowledge Enhanced Visual Semantic Embeddings
We improve zero-shot learning (ZSL) by incorporating common-sense knowledge in DNNs. We propose Common-Sense based Neuro-Symbolic Loss (CSNL) that formulates prior knowledge as novel neuro-symbolic loss functions that regularize visual-semantic embedding. CSNL forces visual features in the VSE to obey common-sense rules relating to hypernyms and attributes. We introduce two key novelties for improved learning: (1) enforcement of rules for a group instead of a single concept to take into account class-wise relationships, and (2) confidence margins inside logical operators that enable implicit curriculum learning and prevent premature overfitting. We evaluate the advantages of incorporating each knowledge source and show consistent gains over prior state-of-art methods in both conventional and generalized ZSL e.g. 11.5%, 5.5%, and 11.6% improvements on AWA2, CUB, and Kinetics respectively.
Code (0)
등록된 구현이 없습니다.
Tasks
Common Sense ReasoningZero-Shot LearningSimilar Papers 제목 키워드 기반
Attend and Enrich: Enhanced Visual Prompt for Zero-Shot Learning
Zero-shot learning (ZSL) endeavors to transfer knowledge from seen categories to recognize unseen categories, which mostly relies on the semantic-visual interactions between image and attribute tokens. Recently, prompt l…
AttributeDomain GeneralizationInstruction FollowingPrompt Engineering+3Knowledge Graph-Based Explainable and Generalized Zero-Shot Semantic Communications
Data-driven semantic communication is based on superficial statistical patterns, thereby lacking interpretability and generalization, especially for applications with the presence of unseen data. To address these challen…
Semantic CommunicationZero-Shot LearningTowards Zero-shot Human-Object Interaction Detection via Vision-Language Integration
Human-object interaction (HOI) detection aims to locate human-object pairs and identify their interaction categories in images. Most existing methods primarily focus on supervised learning, which relies on extensive manu…
DecoderHuman-Object Interaction DetectionLanguage ModelingLanguage Modelling+2Semantic Enhanced Knowledge Graph for Large-Scale Zero-Shot Learning
Zero-Shot Learning has been a highlighted research topic in both vision and language areas. Recently, most existing methods adopt structured knowledge information to model explicit correlations among categories and use d…
Zero-Shot LearningKnowledge-Enhanced Dual-stream Zero-shot Composed Image Retrieval
We study the zero-shot Composed Image Retrieval (ZS-CIR) task, which is to retrieve the target image given a reference image and a description without training on the triplet datasets. Previous works generate pseudo-word…
AttributeImage RetrievalRetrievalTriplet+1