ECOR: Explainable CLIP for Object Recognition
Large Vision Language Models (VLMs), such as CLIP, have significantly contributed to various computer vision tasks, including object recognition and object detection. Their open vocabulary feature enhances their value. However, their black-box nature and lack of explainability in predictions make them less trustworthy in critical domains. Recently, some work has been done to force VLMs to provide reasonable rationales for object recognition, but this often comes at the expense of classification accuracy. In this paper, we first propose a mathematical definition of explainability in the object recognition task based on the joint probability distribution of categories and rationales, then leverage this definition to fine-tune CLIP in an explainable manner. Through evaluations of different datasets, our method demonstrates state-of-the-art performance in explainable classification. Notably, it excels in zero-shot settings, showcasing its adaptability. This advancement improves explainable object recognition, enhancing trust across diverse applications. The code will be made available online upon publication.
Code (0)
등록된 구현이 없습니다.
Tasks
Objectobject-detectionObject DetectionObject RecognitionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Multi-Rationale Explainable Object Recognition via Contrastive Conditional Inference
Explainable object recognition using vision-language models such as CLIP involves predicting accurate category labels supported by rationales that justify the decision-making process. Existing methods typically rely on p…
Object RecognitionUAV-GESTURE: A Dataset for UAV Control and Gesture Recognition
Current UAV-recorded datasets are mostly limited to action recognition and object tracking, whereas the gesture signals datasets were mostly recorded in indoor spaces. Currently, there is no outdoor recorded public video…
Action RecognitionGesture RecognitionObject TrackingTemporal Action LocalizationA ChatGPT Aided Explainable Framework for Zero-Shot Medical Image Diagnosis
Zero-shot medical image classification is a critical process in real-world scenarios where we have limited access to all possible diseases or large-scale annotated data. It involves computing similarity scores between a …
Diagnosticimage-classificationImage ClassificationMedical Image ClassificationA novel Framework for Open-Vocabulary Multi-Object Recognition using CLIP
To address the limitations of existing open-vocabulary object recognition methods, including high system complexity, substantial training costs, and limited generalization capability, this paper proposes a novel Open-Voc…
Object SegmentationObject RecognitionLearning complete and explainable visual representations from itemized text supervision
Training vision models with language supervision enables general and transferable representations. However, many visual domains, especially non-object-centric domains such as medical imaging and remote sensing, contain i…