Model Consistency as a Cheap yet Predictive Proxy for LLM Elo Scores

2025-09-27 · Ashwin Ramaswamy, Nestor Demeure, Ermal Rrapaj arxiv

New large language models (LLMs) are being released every day. Some perform significantly better or worse than expected given their parameter count. Therefore, there is a need for a method to independently evaluate models. The current best way to evaluate a model is to measure its Elo score by comparing it to other models in a series of contests - an expensive operation since humans are ideally required to compare LLM outputs. We observe that when an LLM is asked to judge such contests, the consistency with which it selects a model as the best in a matchup produces a metric that is 91% correlated with its own human-produced Elo score. This provides a simple proxy for Elo scores that can be computed cheaply, without any human data or prior knowledge.

📄 PDF Abstract BibTeX arXiv:2509.23510

Code (0)

등록된 구현이 없습니다.

Sol-Attn: Accelerating Video Generation Inference via On-the-Fly Attention Sparsification

2026-07-27 · Haopeng Li, Yitong Li, Junsong Chen, Tian Ye 외 hf

Diffusion transformers are essential for high-fidelity video generation, but long token sequences make attention a dominant inference bottleneck. Training-free dynamic sparse attention alleviates this bottleneck by compu…

Video Generation

Predicting with Proxies: Transfer Learning in High Dimension

2018-12-28 · Hamsa Bastani

Predictive analytics is increasingly used to guide decision-making in many applications. However, in practice, we often have limited data on the true predictive task of interest, and must instead rely on more abundant da…

Decision Making Transfer Learning Vocal Bursts Intensity Prediction

Best Arm Identification with LLM Judges and Limited Human

2026-01-29 · Ruicheng Ao, Hongyu Chen, Siyang Gao, Hanwei Li 외 arxiv

We study fixed-confidence best-arm identification (BAI) where a cheap but potentially biased proxy (e.g., LLM judge) is available for every sample, while an expensive ground-truth label can only be acquired selectively w…

Semantically-Guided Inference for Conditional Diffusion Models: Enhancing Covariate Consistency in Time Series Forecasting

2025-08-03 · Rui Ding, Hanyang Meng, Zeyang Zhang, Jielong Yang arxiv

Diffusion models have demonstrated strong performance in time series forecasting, yet often suffer from semantic misalignment between generated trajectories and conditioning covariates, especially under complex or multim…

Time Series Forecasting

PACE: A Proxy for Agentic Capability Evaluation

2026-07-02 · Yueqi Song, Lintang Sutawika, Jiarui Liu, Lindia Tjuatja 외 arxiv

Evaluating LLM agents on benchmarks like SWE-Bench and GAIA can be expensive, time-consuming, and requires complex infrastructure. A single evaluation can cost thousands of dollars and take days to complete. In contrast,…

Code Generation