Lightweight reranking for language model generations
Large Language Models (LLMs) can exhibit considerable variation in the quality of their sampled outputs. Reranking and selecting the best generation from the sampled set is a popular way of obtaining strong gains in generation quality. In this paper, we present a novel approach for reranking LLM generations. Unlike other techniques that might involve additional inferences or training a specialized reranker, our approach relies on easy to compute pairwise statistics between the generations that have minimal compute overhead. We show that our approach can be formalized as an extension of self-consistency and analyze its performance in that framework, theoretically as well as via simulations. We show strong improvements for selecting the best k generations for code generation tasks as well as robust improvements for the best generation for the tasks of autoformalization, summarization, and translation. While our approach only assumes black-box access to LLMs, we show that additional access to token probabilities can improve performance even further.
Code (0)
등록된 구현이 없습니다.
Tasks
Code GenerationLanguage ModelingLanguage ModellingmodelRerankingSimilar Papers 제목 키워드 기반
FANTAstic SEquences and Where to Find Them: Faithful and Efficient API Call Generation through State-tracked Constrained Decoding and Reranking
API call generation is the cornerstone of large language models' tool-using ability that provides access to the larger world. However, existing supervised and in-context learning approaches suffer from high training cost…
In-Context LearningLanguage ModelingLanguage ModellingLarge Language Model+1How Good are LLM-based Rerankers? An Empirical Analysis of State-of-the-Art Reranking Models
In this work, we present a systematic and comprehensive empirical evaluation of state-of-the-art reranking methods, encompassing large language model (LLM)-based, lightweight contextual, and zero-shot approaches, with re…
Computational EfficiencyInformation RetrievalHLATR: Enhance Multi-stage Text Retrieval with Hybrid List Aware Transformer Reranking
Deep pre-trained language models (e,g. BERT) are effective at large-scale text retrieval task. Existing text retrieval systems with state-of-the-art performance usually adopt a retrieve-then-reranking architecture due to…
Passage RankingPassage Re-RankingRerankingRetrieval+1Universal Biological Sequence Reranking for Improved De Novo Peptide Sequencing
De novo peptide sequencing is a critical task in proteomics. However, the performance of current deep learning-based methods is limited by the inherent complexity of mass spectrometry data and the heterogeneous distribut…
de novo peptide sequencingRerankingZero-shot GeneralizationInstance-level Image Retrieval using Reranking Transformers
Instance-level image retrieval is the task of searching in a large database for images that match an object in a query image. To address this task, systems usually rely on a retrieval step that uses global image descript…
Image RetrievalRerankingRetrieval