Diagnosing LLM-based Rerankers in Cold-Start Recommender Systems: Coverage, Exposure and Practical Mitigations
Large language models (LLMs) and cross-encoder rerankers have gained attention for improving recommender systems, particularly in cold-start scenarios where user interaction history is limited. However, practical deployment reveals significant performance gaps between LLM-based approaches and simple baselines. This paper presents a systematic diagnostic study of cross-encoder rerankers in cold-start movie recommendation using the Serendipity-2018 dataset. Through controlled experiments with 500 users across multiple random seeds, we identify three critical failure modes: (1) low retrieval coverage in candidate generation (recall@200 = 0.109 vs. 0.609 for baselines), (2) severe exposure bias with rerankers concentrating recommendations on 3 unique items versus 497 for random baseline, and (3) minimal score discrimination between relevant and irrelevant items (mean difference = 0.098, Cohen's d = 0.13). We demonstrate that popularity-based ranking substantially outperforms LLM reranking (HR@10: 0.268 vs. 0.008, p < 0.001), with the performance gap primarily attributable to retrieval stage limitations rather than reranker capacity. Based on these findings, we provide actionable recommendations including hybrid retrieval strategies, candidate pool size optimization, and score calibration techniques. All code, configurations, and experimental results are made available for reproducibility.
Code (0)
등록된 구현이 없습니다.
Tasks
Movie RecommendationSimilar Papers 제목 키워드 기반
Diagnosing and Mitigating Retrieval Bottlenecks in LLM-Based Cold-Start Recommendation
Large language models (LLMs) are increasingly used as rerankers in recommender systems, with the expectation that semantic understanding will help in cold-start and long-tail regimes. We test this assumption with a five-…
Handling Cold-Start Collaborative Filtering with Reinforcement Learning
A major challenge in recommender systems is handling new users, whom are also called $\textit{cold-start}$ users. In this paper, we propose a novel approach for learning an optimal series of questions with which to inter…
Collaborative FilteringRecommendation Systemsreinforcement-learningReinforcement Learning+1Deep Learning to Address Candidate Generation and Cold Start Challenges in Recommender Systems: A Research Survey
Among the machine learning applications to business, recommender systems would take one of the top places when it comes to success and adoption. They help the user in accelerating the process of search while helping busi…
Deep LearningRecommendation Systemsspeech-recognitionSpeech RecognitionDotMat: Solving Cold-start Problem and Alleviating Sparsity Problem for Recommender Systems
Cold-start and sparsity problem are two key intrinsic problems to recommender systems. During the past two decades, researchers and industrial practitioners have spent considerable amount of efforts trying to solve the p…
Recommendation SystemsDropoutNet: Addressing Cold Start in Recommender Systems
Latent models have become the default choice for recommender systems due to their performance and scalability. However, research in this area has primarily focused on modeling user-item interactions, and few latent mode…
Recommendation Systems