In-context Learning with Retrieved Demonstrations for Language Models: A Survey
Language models, especially pre-trained large language models, have showcased remarkable abilities as few-shot in-context learners (ICL), adept at adapting to new tasks with just a few demonstrations in the input context. However, the model's ability to perform ICL is sensitive to the choice of the few-shot demonstrations. Instead of using a fixed set of demonstrations, one recent development is to retrieve demonstrations tailored to each input query. The implementation of demonstration retrieval is relatively straightforward, leveraging existing databases and retrieval systems. This not only improves the efficiency and scalability of the learning process but also has been shown to reduce biases inherent in manual example selection. In light of the encouraging results and growing research in ICL with retrieved demonstrations, we conduct an extensive review of studies in this area. In this survey, we discuss and compare different design choices for retrieval models, retrieval training procedures, and inference algorithms.
Code (0)
등록된 구현이 없습니다.
Tasks
In-Context LearningRetrievalMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
DemoRank: Selecting Effective Demonstrations for Large Language Models in Ranking Task
Recently, there has been increasing interest in applying large language models (LLMs) as zero-shot passage rankers. However, few studies have explored how to select appropriate in-context demonstrations for the passage r…
In-Context LearningPassage RankingMixture of Demonstrations for Textual Graph Understanding and Question Answering
Textual graph-based retrieval-augmented generation (GraphRAG) has emerged as a powerful paradigm for enhancing large language models (LLMs) in domain-specific question answering. While existing approaches primarily focus…
Question AnsweringDr.ICL: Demonstration-Retrieved In-context Learning
In-context learning (ICL), teaching a large language model (LLM) to perform a task with few-shot demonstrations rather than adjusting the model parameters, has emerged as a strong paradigm for using LLMs. While early stu…
In-Context LearningLanguage ModelingLanguage ModellingLarge Language Model+1DRUM: Learning Demonstration Retriever for Large MUlti-modal Models
Recently, large language models (LLMs) have demonstrated impressive capabilities in dealing with new tasks with the help of in-context learning (ICL). In the study of Large Vision-Language Models (LVLMs), when implementi…
In-Context LearningRetrievalBAGEL: Bootstrapping Agents by Guiding Exploration with Language
Following natural language instructions by executing actions in digital environments (e.g. web-browsers and REST APIs) is a challenging task for language model (LM) agents. Unfortunately, LM agents often fail to generali…
In-Context LearningLanguage ModelingLanguage Modelling