Convolutional Visual Prompt for Robust Visual Perception
Vision models are often vulnerable to out-of-distribution (OOD) samples without adapting. While visual prompts offer a lightweight method of input-space adaptation for large-scale vision models, they rely on a high-dimensional additive vector and labeled data. This leads to overfitting when adapting models in a self-supervised test-time setting without labels. We introduce convolutional visual prompts (CVP) for label-free test-time adaptation for robust visual perception. The structured nature of CVP demands fewer trainable parameters, less than 1\% compared to standard visual prompts, combating overfitting. Extensive experiments and analysis on a wide variety of OOD visual perception tasks show that our approach is effective, improving robustness by up to 5.87% over several large-scale models.
Code (0)
등록된 구현이 없습니다.
Tasks
Test-time AdaptationSimilar Papers 제목 키워드 기반
Joint Visual and Text Prompting for Improved Object-Centric Perception with Multimodal Large Language Models
Multimodal Large Language Models (MLLMs) such as GPT-4V and Gemini Pro face challenges in achieving human-level perception in Visual Question Answering (VQA), particularly in object-oriented perception tasks which demand…
MMEObjectQuestion AnsweringVisual Question Answering+1Visual Prompt Discovery via Semantic Exploration
LVLMs encounter significant challenges in image understanding and visual reasoning, leading to critical perception failures. Visual prompts, which incorporate image manipulation code, have shown promising potential in mi…
Image ManipulationVisual ReasoningVisual Exemplar Driven Task-Prompting for Unified Perception in Autonomous Driving
Multi-task learning has emerged as a powerful paradigm to solve a range of tasks simultaneously with good efficiency in both computation resources and inference time. However, these algorithms are designed for different …
Autonomous DrivingLane DetectionMulti-Task Learningobject-detection+2Visual Prompting in Multimodal Large Language Models: A Survey
Multimodal large language models (MLLMs) equip pre-trained large-language models (LLMs) with visual capabilities. While textual prompting in LLMs has been widely studied, visual prompting has emerged for more fine-graine…
In-Context LearningPrompt LearningSurveyVisual Grounding+1MR-MLLM: Mutual Reinforcement of Multimodal Comprehension and Vision Perception
In recent years, multimodal large language models (MLLMs) have shown remarkable capabilities in tasks like visual question answering and common sense reasoning, while visual perception models have made significant stride…
Common Sense ReasoningLanguage ModellingLarge Language ModelMultimodal Large Language Model+4