Finding Visual Task Vectors
Visual Prompting is a technique for teaching models to perform a visual task via in-context examples, without any additional training. In this work, we analyze the activations of MAE-VQGAN, a recent Visual Prompting model, and find task vectors, activations that encode task-specific information. Equipped with this insight, we demonstrate that it is possible to identify the task vectors and use them to guide the network towards performing different tasks without providing any input-output examples. To find task vectors, we compute the average intermediate activations per task and use the REINFORCE algorithm to search for the subset of task vectors. The resulting task vectors guide the model towards performing a task better than the original model without the need for input-output examples.
Code (1)
Tasks
Visual PromptingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Understanding Task Vectors in In-Context Learning: Emergence, Functionality, and Limitations
Task vectors offer a compelling mechanism for accelerating inference in in-context learning (ICL) by distilling task-specific information into a single, reusable representation. Despite their empirical success, the under…
In-Context LearningTripletBrain-aligning of semantic vectors improves neural decoding of visual stimuli
The development of algorithms to accurately decode of neural information is a long-standing effort in the field of neuroscience. Brain decoding is typically employed by training machine learning models to map neural data…
Brain DecodingRepresentation LearningDynamic Multimodal Activation Steering for Hallucination Mitigation in Large Vision-Language Models
Large Vision-Language Models (LVLMs) exhibit outstanding performance on vision-language tasks but struggle with hallucination problems. Through in-depth analysis of LVLM activation patterns, we reveal two key findings: 1…
Semantic SimilarityGrounded Compositional Semantics for Finding and Describing Images with Sentences
Previous work on Recursive Neural Networks (RNNs) shows that these models can produce compositional feature vectors for accurately representing and classifying sentences or images. However, the sentence vectors of previo…
SentenceVISOR++: Universal Visual Inputs based Steering for Large Vision Language Models
As Vision Language Models (VLMs) are deployed across safety-critical applications, understanding and controlling their behavioral patterns has become increasingly important. Existing behavioral control methods face signi…