Efficient Evaluation of Large Language Models via Collaborative Filtering
With the development of Large Language Models (LLMs), numerous benchmarks have been proposed to measure and compare the capabilities of different LLMs. However, evaluating LLMs is costly due to the large number of test instances and their slow inference speed. In this paper, we aim to explore how to efficiently estimate a model's real performance on a given benchmark based on its evaluation results on a small number of instances sampled from the benchmark. Inspired by Collaborative Filtering (CF) in Recommendation Systems (RS), we treat LLMs as users and test instances as items and propose a two-stage method. In the first stage, we treat instance selection as recommending products to users to choose instances that can easily distinguish model performance. In the second stage, we see performance prediction as rating prediction problem in RS to predict the target LLM's behavior on unselected instances. Experiments on multiple LLMs and datasets imply that our method can accurately estimate the target model's performance while largely reducing its inference overhead.
Code (0)
등록된 구현이 없습니다.
Tasks
Collaborative FilteringRecommendation SystemsSimilar Papers 제목 키워드 기반
Enhanced Recommendation Combining Collaborative Filtering and Large Language Models
With the advent of the information explosion era, the importance of recommendation systems in various applications is increasingly significant. Traditional collaborative filtering algorithms are widely used due to their …
Collaborative FilteringDiversityNatural Language UnderstandingRecommendation SystemsReducing offline evaluation bias of collaborative filtering algorithms
Recommendation systems have been integrated into the majority of large online systems to filter and rank information according to user profiles. It thus influences the way users interact with the system and, as a consequ…
Collaborative FilteringRecommendation SystemsLarge Language Models Enhanced Collaborative Filtering
Recent advancements in Large Language Models (LLMs) have attracted considerable interest among researchers to leverage these models to enhance Recommender Systems (RSs). Existing work predominantly utilizes LLMs to gener…
Collaborative FilteringIn-Context LearningRecommendation SystemsWorld KnowledgeOptimization and Scalability of Collaborative Filtering Algorithms in Large Language Models
With the rapid development of large language models (LLMs) and the growing demand for personalized content, recommendation systems have become critical in enhancing user experience and driving engagement. Collaborative f…
Collaborative FilteringComputational EfficiencyModel CompressionRecommendation SystemsMolar: Multimodal LLMs with Collaborative Filtering Alignment for Enhanced Sequential Recommendation
Sequential recommendation (SR) systems have evolved significantly over the past decade, transitioning from traditional collaborative filtering to deep learning approaches and, more recently, to large language models (LLM…
Collaborative FilteringSequential Recommendation