Real Classification by Description: Extending CLIP's Limits of Part Attributes Recognition
In this study, we define and tackle zero shot "real" classification by description, a novel task that evaluates the ability of Vision-Language Models (VLMs) like CLIP to classify objects based solely on descriptive attributes, excluding object class names. This approach highlights the current limitations of VLMs in understanding intricate object descriptions, pushing these models beyond mere object recognition. To facilitate this exploration, we introduce a new challenge and release description data for six popular fine-grained benchmarks, which omit object names to encourage genuine zero-shot learning within the research community. Additionally, we propose a method to enhance CLIP's attribute detection capabilities through targeted training using ImageNet21k's diverse object categories, paired with rich attribute descriptions generated by large language models. Furthermore, we introduce a modified CLIP architecture that leverages multiple resolutions to improve the detection of fine-grained part attributes. Through these efforts, we broaden the understanding of part-attribute recognition in CLIP, improving its performance in fine-grained classification tasks across six popular benchmarks, as well as in the PACO dataset, a widely used benchmark for object-attribute recognition. Code is available at: https://github.com/ethanbar11/grounding_ge_public.
Code (0)
등록된 구현이 없습니다.
Tasks
AttributeDescriptiveObjectObject RecognitionZero-Shot LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
TalkCLIP: Talking Head Generation with Text-Guided Expressive Speaking Styles
Audio-driven talking head generation has drawn growing attention. To produce talking head videos with desired facial expressions, previous methods rely on extra reference videos to provide expression information, which m…
2D Semantic Segmentation task 3 (25 classes)Talking Head GenerationEmoCLIP: A Vision-Language Method for Zero-Shot Video Facial Expression Recognition
Facial Expression Recognition (FER) is a crucial task in affective computing, but its conventional focus on the seven basic emotions limits its applicability to the complex and expanding emotional spectrum. To address th…
Facial Expression RecognitionFacial Expression Recognition (FER)Language Modellingzero-shot-classification+2Human Pose Descriptions and Subject-Focused Attention for Improved Zero-Shot Transfer in Human-Centric Classification Tasks
We present a novel LLM-based pipeline for creating contextual descriptions of human body poses in images using only auxiliary attributes. This approach facilitates the creation of the MPII Pose Descriptions dataset, whic…
Activity RecognitionAge ClassificationEmotion RecognitionEnhanced Motion-Text Alignment for Image-to-Video Transfer Learning
Extending large image-text pre-trained models (e.g. CLIP) for video understanding has made significant advancements. To enable the capability of CLIP to perceive dynamic information in videos existing works are dedic…
Transfer LearningVideo UnderstandingLMM-Regularized CLIP Embeddings for Image Classification
In this paper we deal with image classification tasks using the powerful CLIP vision-language model. Our goal is to advance the classification performance using the CLIP's image encoder, by proposing a novel Large Multim…
Classificationimage-classificationImage ClassificationLanguage Modeling+1