Learning to Select Visual In-Context Demonstrations

2026-03-24 · Eugene Lee, Yu-Chi Lin, Jiajie Diao arxiv

Multimodal Large Language Models (MLLMs) adapt to visual tasks via in-context learning (ICL), which relies heavily on demonstration quality. The dominant demonstration selection strategy is unsupervised k-Nearest Neighbor (kNN) search. While simple, this similarity-first approach is sub-optimal for complex factual regression tasks; it selects redundant examples that fail to capture the task's full output range. We reframe selection as a sequential decision-making problem and introduce Learning to Select Demonstrations (LSD), training a Reinforcement Learning agent to construct optimal demonstration sets. Using a Dueling DQN with a query-centric Transformer Decoder, our agent learns a policy that maximizes MLLM downstream performance. Evaluating across five visual regression benchmarks, we uncover a crucial dichotomy: while kNN remains optimal for subjective preference tasks, LSD significantly outperforms baselines on objective, factual regression tasks. By balancing visual relevance with diversity, LSD better defines regression boundaries, illuminating when learned selection is strictly necessary for visual ICL.

📄 PDF Abstract BibTeX arXiv:2603.26775

Code (0)

등록된 구현이 없습니다.

Tasks

Reinforcement Learning

DRUM: Learning Demonstration Retriever for Large MUlti-modal Models

2024-12-10 · Ellen Yi-Ge, Jiechao Gao, Wei Han, Wei Zhu

Recently, large language models (LLMs) have demonstrated impressive capabilities in dealing with new tasks with the help of in-context learning (ICL). In the study of Large Vision-Language Models (LVLMs), when implementi…

In-Context Learning Retrieval

In-Context Demonstration Selection with Cross Entropy Difference

2023-05-24 · Dan Iter, Reid Pryzant, Ruochen Xu, Shuohang Wang 외

Large language models (LLMs) can use in-context demonstrations to improve performance on zero-shot tasks. However, selecting the best in-context examples is challenging because model performance can vary widely depending…

Language Modeling Language Modelling Text Generation

Provoking Multi-modal Few-Shot LVLM via Exploration-Exploitation In-Context Learning

2025-06-11 · CVPR 2025 1 · Cheng Chen, Yunpeng Zhai, Yifan Zhao, Jinyang Gao 외

In-context learning (ICL), a predominant trend in instruction learning, aims at enhancing the performance of large language models by providing clear task guidance and examples, improving their capability in task underst…

In-Context Learning Question Answering Visual Question Answering Visual Question Answering (VQA)

Learning to Select In-Context Demonstration Preferred by Large Language Model

2025-05-26 · Zheng Zhang, Shaocheng Lan, Lei Song, Jiang Bian 외

In-context learning (ICL) enables large language models (LLMs) to adapt to new tasks during inference using only a few demonstrations. However, ICL performance is highly dependent on the selection of these demonstrations…

In-Context Learning Language Modeling Language Modelling Large Language Model+2

Diverse Demonstrations Improve In-context Compositional Generalization

2022-12-13 · Itay Levy, Ben Bogin, Jonathan Berant

In-context learning has shown great success in i.i.d semantic parsing splits, where the training and test sets are drawn from the same distribution. In this setup, models are typically prompted with demonstrations that a…

In-Context Learning Semantic Parsing