Large-Scale Few-Shot Learning via Multi-Modal Knowledge Discovery
Large-scale few-shot learning aims at identifying hundreds of novel object categories where each category has only a few samples. It is a challenging problem since (1) the identifying process is susceptible to over-fitting with limited samples of an object, and (2) the sample imbalance between a base (known knowledge) category and a novel category is easy to bias the recognition results. To solve these problems, we propose a method based on multi-modal knowledge discovery. First, we use the visual knowledge to help the feature extractors focus on different visual parts. Second, we design a classifier to learn the distribution over all categories. In the second stage, we develop three schemes to minimize the prediction error and balance the training procedure: (1) Hard labels are used to provide precise supervision. (2) Semantic textual knowledge is utilized as weak supervision to find the potential relations between the novel and the base categories. (3) An imbalance control is presented from the data distribution to alleviate the recognition bias towards the base categories. We apply our method on three benchmark datasets, and it achieves state-of-the-art performances in all the experiments.
Code (0)
등록된 구현이 없습니다.
Tasks
Few-Shot LearningSimilar Papers 제목 키워드 기반
Recognizing Unseen Objects via Multimodal Intensive Knowledge Graph Propagation
Zero-Shot Learning (ZSL), which aims at automatically recognizing unseen objects, is a promising learning paradigm to understand new real-world knowledge for machines continuously. Recently, the Knowledge Graph (KG) has …
AttributeKnowledge GraphsWorld Knowledgezero-shot-classification+1Connecting Giants: Synergistic Knowledge Transfer of Large Multimodal Models for Few-Shot Learning
Few-shot learning (FSL) addresses the challenge of classifying novel classes with limited training samples. While some methods leverage semantic knowledge from smaller-scale models to mitigate data scarcity, these approa…
Few-Shot LearningEnabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation
The recent large-scale vision-language pre-training (VLP) of dual-stream architectures (e.g., CLIP) with a tremendous amount of image-text pair data, has shown its superiority on various multimodal alignment tasks. Despi…
Image CaptioningKnowledge DistillationLanguage ModelingLanguage Modelling+4Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation
The recent large-scale vision-language pre-training (VLP) of dual-stream architectures (e.g., CLIP) with a tremendous amount of image-text pair data, has shown its superiority on various multimodal alignment tasks. Despi…
Image CaptioningKnowledge DistillationLanguage ModelingLanguage Modelling+4Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language
Large pretrained (e.g., "foundation") models exhibit distinct capabilities depending on the domain of data they are trained on. While these domains are generic, they may only barely overlap. For example, visual-language …
DiversityImage CaptioningMultimodal ReasoningRetrieval+4