Sample Efficient Demonstration Selection for In-Context Learning

2025-06-10 · Kiran Purohit, V Venktesh, Sourangshu Bhattacharya, Avishek Anand

The in-context learning paradigm with LLMs has been instrumental in advancing a wide range of natural language processing tasks. The selection of few-shot examples (exemplars / demonstration samples) is essential for constructing effective prompts under context-length budget constraints. In this paper, we formulate the exemplar selection task as a top-m best arms identification problem. A key challenge in this setup is the exponentially large number of arms that need to be evaluated to identify the m-best arms. We propose CASE (Challenger Arm Sampling for Exemplar selection), a novel sample-efficient selective exploration strategy that maintains a shortlist of "challenger" arms, which are current candidates for the top-m arms. In each iteration, only one of the arms from this shortlist or the current topm set is pulled, thereby reducing sample complexity and, consequently, the number of LLM evaluations. Furthermore, we model the scores of exemplar subsets (arms) using a parameterized linear scoring function, leading to stochastic linear bandits setting. CASE achieves remarkable efficiency gains of up to 7x speedup in runtime while requiring 7x fewer LLM calls (87% reduction) without sacrificing performance compared to state-of-the-art exemplar selection methods. We release our code and data at https://github.com/kiranpurohit/CASE

📄 PDF Abstract BibTeX arXiv:2506.08607

Code (1)

kiranpurohit/case 공식 구현

Tasks

In-Context Learning

Methods 이 논문이 사용한 방법론

SET Dynamic Sparse Training method where weight mask is updated randomly periodically

In-Context Learning with Iterative Demonstration Selection

2023-10-15 · Chengwei Qin, Aston Zhang, Chen Chen, Anirudh Dagar 외

Spurred by advancements in scale, large language models (LLMs) have demonstrated strong few-shot learning ability via in-context learning (ICL). However, the performance of ICL has been shown to be highly sensitive to th…

Few-Shot Learning In-Context Learning Question Answering Sentiment Analysis+1

Easier to Judge than to Find: Predicting In-Context Learning Success for Demonstration Selection

2026-05-18 · Haochun Wang, Chaofen Yang, Jiatong Liu, Jingbo Wang 외 arxiv

In-context learning (ICL) is highly sensitive to which demonstrations appear in the prompt, but selecting them is expensive because the space of possible demonstration contexts and combinations is enormous. We argue that…

In-Context Learning Demonstration Selection via Influence Analysis

2024-02-19 · Vinay M. S., Minh-Hao Van, Xintao Wu

Large Language Models (LLMs) have showcased their In-Context Learning (ICL) capabilities, enabling few-shot learning without the need for gradient updates. Despite its advantages, the effectiveness of ICL heavily depends…

Few-Shot Learning In-Context Learning

Efficient and Effective In-context Demonstration Selection with Coreset

2025-11-12 · Zihua Wang, Jiarui Wang, Haiyang Xu, Ming Yan 외 arxiv

In-context learning (ICL) has emerged as a powerful paradigm for Large Visual Language Models (LVLMs), enabling them to leverage a few examples directly from input contexts. However, the effectiveness of this approach is…

Curriculum Demonstration Selection for In-Context Learning

2024-11-27 · Duc Anh Vu, Nguyen Tran Cong Duy, Xiaobao Wu, Hoang Minh Nhat 외

Large Language Models (LLMs) have shown strong in-context learning (ICL) abilities with a few demonstrations. However, one critical challenge is how to select demonstrations to elicit the full potential of LLMs. In this …