In-Context Example Selection via Similarity Search Improves Low-Resource Machine Translation
The ability of generative large language models (LLMs) to perform in-context learning has given rise to a large body of research into how best to prompt models for various natural language processing tasks. In this paper, we focus on machine translation (MT), a task that has been shown to benefit from in-context translation examples. However no systematic studies have been published on how best to select examples, and mixed results have been reported on the usefulness of similarity-based selection over random selection. We provide a study covering multiple LLMs and multiple in-context example retrieval strategies, comparing multilingual sentence embeddings. We cover several language directions, representing different levels of language resourcedness (English into French, German, Swahili and Wolof). Contrarily to previously published results, we find that sentence embedding similarity can improve MT, especially for low-resource language directions, and discuss the balance between selection pool diversity and quality. We also highlight potential problems with the evaluation of LLM-based MT and suggest a more appropriate evaluation protocol, adapting the COMET metric to the evaluation of LLMs. Code and outputs are freely available at https://github.com/ArmelRandy/ICL-MT.
Code (1)
Tasks
DiversityIn-Context LearningMachine TranslationSentenceSentence EmbeddingSentence-EmbeddingSentence EmbeddingsTranslationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Going Beyond Word Matching: Syntax Improves In-context Example Selection for Machine Translation
In-context learning (ICL) is the trending prompting strategy in the era of large language models (LLMs), where a few examples are demonstrated to evoke LLMs' power for a given task. How to select informative examples rem…
In-Context LearningMachine TranslationTranslationExploring Example Selection for Few-shot Text-to-SQL Semantic Parsing
We study example selection methods for few-shot text-to-SQL tasks with unseen databases. Annotating natural language questions with corresponding SQL queries is expensive, but we can use abundant unlabeled questions to e…
Few-Shot LearningIn-Context LearningSemantic ParsingText to SQL+1Exploring the Role of Diversity in Example Selection for In-Context Learning
In-Context Learning (ICL) has gained prominence due to its ability to perform tasks without requiring extensive training data and its robustness to noisy labels. A typical ICL workflow involves selecting localized exampl…
DiversityIn-Context LearningRerankingFailures Are the Stepping Stones to Success: Enhancing Few-Shot In-Context Learning by Leveraging Negative Samples
Large Language Models exhibit powerful few-shot in-context learning (ICL) capabilities, but the performance is highly sensitive to provided examples. Recent research has focused on retrieving corresponding examples for e…
Semantic SimilarityVisual Prompt Selection for In-Context Learning Segmentation
As a fundamental and extensively studied task in computer vision, image segmentation aims to locate and identify different semantic concepts at the pixel level. Recently, inspired by In-Context Learning (ICL), several ge…
DiversityImage SegmentationIn-Context LearningSegmentation+1