Ranked List Truncation for Large Language Model-based Re-Ranking
We study ranked list truncation (RLT) from a novel "retrieve-then-re-rank" perspective, where we optimize re-ranking by truncating the retrieved list (i.e., trim re-ranking candidates). RLT is crucial for re-ranking as it can improve re-ranking efficiency by sending variable-length candidate lists to a re-ranker on a per-query basis. It also has the potential to improve re-ranking effectiveness. Despite its importance, there is limited research into applying RLT methods to this new perspective. To address this research gap, we reproduce existing RLT methods in the context of re-ranking, especially newly emerged large language model (LLM)-based re-ranking. In particular, we examine to what extent established findings on RLT for retrieval are generalizable to the "retrieve-then-re-rank" setup from three perspectives: (i) assessing RLT methods in the context of LLM-based re-ranking with lexical first-stage retrieval, (ii) investigating the impact of different types of first-stage retrievers on RLT methods, and (iii) investigating the impact of different types of re-rankers on RLT methods. We perform experiments on the TREC 2019 and 2020 deep learning tracks, investigating 8 RLT methods for pipelines involving 3 retrievers and 2 re-rankers. We reach new insights into RLT methods in the context of re-ranking.
Code (1)
Tasks
Language ModelingLanguage ModellingLarge Language ModelRe-RankingRetrievalSimilar Papers 제목 키워드 기반
List-aware Reranking-Truncation Joint Model for Search and Retrieval-augmented Generation
The results of information retrieval (IR) are usually presented in the form of a ranked list of candidate documents, such as web search for humans and retrieval-augmented generation for large language models (LLMs). List…
Information RetrievalLearning-To-RankMisinformationReranking+2Choppy: Cut Transformer For Ranked List Truncation
Work in information retrieval has traditionally focused on ranking and relevance: given a query, return some number of results ordered by relevance to the user. However, the problem of determining how many results to ret…
Information RetrievalRetrievalSurprise: Result List Truncation via Extreme Value Theory
Work in information retrieval has largely been centered around ranking and relevance: given a query, return some number of results ordered by relevance to the user. The problem of result list truncation, or where to trun…
Information RetrievalRetrievalTwo-sample testingAn Empirical Analysis of the Effect of Ballot Truncation on Ranked-Choice Electoral Outcomes
In ranked-choice elections voters cast preference ballots which provide a voter's ranking of the candidates. The method of ranked-choice voting (RCV) chooses a winner by using voter preferences to simulate a series of ru…
Statistical Models of Top-$k$ Partial Orders
In many contexts involving ranked preferences, agents submit partial orders over available alternatives. Statistical models often treat these as marginal in the space of total orders, but this approach overlooks informat…