Comparative Analysis of Demonstration Selection Algorithms for LLM In-Context Learning
In-context learning can help Large Language Models (LLMs) to adapt new tasks without additional training. However, this performance heavily depends on the quality of the demonstrations, driving research into effective demonstration selection algorithms to optimize this process. These algorithms assist users in selecting the best $k$ input-label pairs (demonstration examples) based on a given test input, enabling LLMs to in-context learn the relationship between the provided examples and the test inputs. Despite all the proposed demonstration selection algorithms, their efficiency and effectiveness remain unclear. This lack of clarity make it difficult to apply these algorithms in real-world scenarios and poses challenges for future research aimed at developing improved methods. This paper revisits six proposed algorithms, evaluating them on five datasets from both efficiency and effectiveness perspectives. Our experiments reveal significant variations in algorithm performance across different tasks, with some methods struggling to outperform random selection in certain scenarios. We also find that increasing the number of demonstrations does not always lead to better performance, and that there are often trade-offs between accuracy and computational efficiency. Our code is available at https://github.com/Tizzzzy/Demonstration_Selection_Overview.
Code (1)
Tasks
Computational EfficiencyIn-Context LearningSimilar Papers 제목 키워드 기반
A Comparative Study of Demonstration Selection for Practical Large Language Models-based Next POI Prediction
This paper investigates demonstration selection strategies for predicting a user's next point-of-interest (POI) using large language models (LLMs), aiming to accurately forecast a user's subsequent location based on hist…
Active Learning Principles for In-Context Learning with Large Language Models
The remarkable advancements in large language models (LLMs) have significantly enhanced the performance in few-shot learning settings. By using only a small number of labeled examples, referred to as demonstrations, LLMs…
Active LearningDiversityFew-Shot LearningIn-Context LearningLearn to Select: Exploring Label Distribution Divergence for In-Context Demonstration Selection in Text Classification
In-context learning (ICL) for text classification, which uses a few input-label demonstrations to describe a task, has demonstrated impressive performance on large language models (LLMs). However, the selection of in-con…
Semantic SimilarityText ClassificationUnifying Demonstration Selection and Compression for In-Context Learning
In-context learning (ICL) facilitates large language models (LLMs) exhibiting spectacular emergent capabilities in various scenarios. Unfortunately, introducing demonstrations easily makes the prompt length explode, brin…
In-Context LearningResponse GenerationSemantic SimilaritySemantic Textual SimilarityIn-Context Learning Demonstration Selection via Influence Analysis
Large Language Models (LLMs) have showcased their In-Context Learning (ICL) capabilities, enabling few-shot learning without the need for gradient updates. Despite its advantages, the effectiveness of ICL heavily depends…
Few-Shot LearningIn-Context Learning