Papers Visual Prompting
“Visual Prompting” 태그가 달린 논문 127편 · 필터 해제
Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Want
The interaction between humans and artificial intelligence (AI) is a crucial factor that reflects the effectiveness of multimodal large language models (MLLMs). However, current MLLMs primarily focus on image-level compr…
Instruction FollowingLanguage ModellingLarge Language Modelmultimodal interaction+4Explore until Confident: Efficient Exploration for Embodied Question Answering
We consider the problem of Embodied Question Answering (EQA), which refers to settings where an embodied agent such as a robot needs to actively explore an environment to gather information until it is confident about th…
Conformal PredictionEfficient ExplorationEmbodied Question AnsweringQuestion Answering+1On the low-shot transferability of [V]-Mamba
The strength of modern large-scale neural networks lies in their ability to efficiently adapt to new tasks with few examples. Although extensive research has investigated the transferability of Vision Transformers (ViTs)…
Few-Shot LearningMambaTransfer LearningVisual PromptingMOKA: Open-World Robotic Manipulation through Mark-Based Visual Prompting
Open-world generalization requires robotic systems to have a profound understanding of the physical world and the user command to solve diverse and complex tasks. While the recent advancement in vision-language models (V…
In-Context LearningObject RearrangementQuestion AnsweringVisual Prompting+1Tumor segmentation on whole slide images: training or prompting?
Tumor segmentation stands as a pivotal task in cancer diagnosis. Given the immense dimensions of whole slide images (WSI) in histology, deep learning approaches for WSI classification mainly operate at patch-wise or supe…
Computational EfficiencySegmentationSemantic SegmentationTumor Segmentation+2Scaffolding Coordinates to Promote Vision-Language Coordination in Large Multi-Modal Models
State-of-the-art Large Multi-Modal Models (LMMs) have demonstrated exceptional capabilities in vision-language tasks. Despite their advanced functionalities, the performances of LMMs are still limited in challenging scen…
Visual PromptingPIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs
Vision language models (VLMs) have shown impressive capabilities across a variety of tasks, from logical reasoning to visual understanding. This opens the door to richer interaction with the world, for example robotic co…
Instruction FollowingLogical ReasoningQuestion AnsweringSpatial Reasoning+2Tune-An-Ellipse: CLIP Has Potential to Find What You Want
Visual prompting of large vision language models such as CLIP exhibits intriguing zero-shot capabilities. A manually drawn red circle commonly used for highlighting can guide CLIP's attention to the surrounding regio…
ObjectReferring ExpressionReferring Expression ComprehensionVisual PromptingGenerative Multimodal Models are In-Context Learners
The human ability to easily solve multimodal tasks in context (i.e., with only a few demonstrations or simple instructions), is what current multimodal systems have largely struggled to imitate. In this work, we demonstr…
In-Context LearningPersonalized Image GenerationQuestion AnsweringVisual Prompting+1LaViP:Language-Grounded Visual Prompts
We introduce a language-grounded visual prompting method to adapt the visual encoder of vision-language models for downstream tasks. By capitalizing on language integration, we devise a parameter-efficient strategy to ad…
Few-Shot LearningTransfer LearningVisual Prompting3DAxiesPrompts: Unleashing the 3D Spatial Task Capabilities of GPT-4V
In this work, we present a new visual prompting method called 3DAxiesPrompts (3DAP) to unleash the capabilities of GPT-4V in performing 3D spatial tasks. Our investigation reveals that while GPT-4V exhibits proficiency i…
3D Object Detectionobject-detectionObject DetectionVisual PromptingTokenize Anything via Prompting
We present a unified, promptable model capable of simultaneously segmenting, recognizing, and captioning anything. Unlike SAM, we aim to build a versatile region representation in the wild via visual prompting. To achiev…
DecoderVisual PromptingEZ-CLIP: Efficient Zeroshot Video Action Recognition
Recent advancements in large-scale pre-training of visual-language models on paired image-text data have demonstrated impressive generalization capabilities for zero-shot tasks. Building on this success, efforts have bee…
Action RecognitionGPUTemporal Action LocalizationVisual Prompting+2ViscoNet: Bridging and Harmonizing Visual and Textual Conditioning for ControlNet
This paper introduces ViscoNet, a novel one-branch-adapter architecture for concurrent spatial and visual conditioning. Our lightweight model requires trainable parameters and dataset size multiple orders of magnitude sm…
Image GenerationPerson Re-IdentificationVirtual Try-onVisual PromptingVisual Prompting Upgrades Neural Network Sparsification: A Data-Model Perspective
The rapid development of large-scale deep learning models questions the affordability of hardware platforms, which necessitates the pruning to reduce their computational and memory footprints. Sparse neural networks as t…
Image ClassificationVisual PromptingViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts
While existing large vision-language multimodal models focus on whole image understanding, there is a prominent gap in achieving region-specific comprehension. Current approaches that use textual coordinates or spatial e…
Visual Commonsense ReasoningVisual PromptingVisual In-Context Prompting
In-context prompting in large language models (LLMs) has become a prevalent approach to improve zero-shot capabilities, but this idea is less explored in the vision domain. Existing visual prompting methods focus on refe…
DecoderSegmentationVisual PromptingT-Rex: Counting by Visual Prompting
We introduce T-Rex, an interactive object counting model designed to first detect and then count any objects. We formulate object counting as an open-set object detection task with the integration of visual prompts. User…
ObjectObject Countingobject-detectionObject Detection+2GeoSAM: Fine-tuning SAM with Multi-Modal Prompts for Mobility Infrastructure Segmentation
In geographical image segmentation, performance is often constrained by the limited availability of training data and a lack of generalizability, particularly for segmenting mobility infrastructure such as roads, sidewal…
Image SegmentationLarge Language ModelSegmentationSemantic Segmentation+3Towards Robust and Accurate Visual Prompting
Visual prompting, an efficient method for transfer learning, has shown its potential in vision tasks. However, previous works focus exclusively on VP from standard source models, it is still unknown how it performs under…
Adversarial RobustnessTransfer LearningVisual Prompting