ET tu, CLIP? Addressing Common Object Errors for Unseen Environments
We introduce a simple method that employs pre-trained CLIP encoders to enhance model generalization in the ALFRED task. In contrast to previous literature where CLIP replaces the visual encoder, we suggest using CLIP as an additional module through an auxiliary object detection objective. We validate our method on the recently proposed Episodic Transformer architecture and demonstrate that incorporating CLIP improves task performance on the unseen validation set. Additionally, our analysis results support that CLIP especially helps with leveraging object descriptions, detecting small objects, and interpreting rare words.
Code (0)
등록된 구현이 없습니다.
Tasks
Objectobject-detectionObject DetectionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
The impact of Compositionality in Zero-shot Multi-label action recognition for Object-based tasks
Addressing multi-label action recognition in videos represents a significant challenge for robotic applications in dynamic environments, especially when the robot is required to cooperate with humans in tasks that involv…
Action RecognitionAction Recognition In Videosimage-classificationImage Classification+2CLIP4HOI: Towards Adapting CLIP for Practical Zero-Shot HOI Detection
Zero-shot Human-Object Interaction (HOI) detection aims to identify both seen and unseen HOI categories. A strong zero-shot HOI detector is supposed to be not only capable of discriminating novel interactions but also ro…
CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentation
Open-vocabulary semantic segmentation presents the challenge of labeling each pixel within an image based on a wide range of text descriptions. In this work, we introduce a novel cost-based approach to adapt vision-langu…
Image SegmentationOpen Vocabulary Semantic SegmentationOpen-Vocabulary Semantic SegmentationSegmentation+2Unveiling Glitches: A Deep Dive into Image Encoding Bugs within CLIP
Understanding the limitations and weaknesses of state-of-the-art models in artificial intelligence is crucial for their improvement and responsible application. In this research, we focus on CLIP, a model renowned for it…
HallucinationImage ComprehensionSeeing the Unseen: Visual Common Sense for Semantic Placement
Computer vision tasks typically involve describing what is present in an image (e.g. classification, detection, segmentation, and captioning). We study a visual common sense task that requires understanding what is not p…
Common Sense ReasoningImage DescriptionObject