ExAnte: A Benchmark for Ex-Ante Inference in Large Language Models
Large language models (LLMs) face significant challenges in ex-ante reasoning, where analysis, inference, or predictions must be made without access to information from future events. Even with explicit prompts enforcing temporal cutoffs, LLMs often generate outputs influenced by internalized knowledge of events beyond the specified cutoff. This paper introduces a novel task and benchmark designed to evaluate the ability of LLMs to reason while adhering to such temporal constraints. The benchmark includes a variety of tasks: stock prediction, Wikipedia event prediction, scientific publication prediction, and Question Answering (QA), designed to assess factual knowledge under temporal cutoff constraints. We use leakage rate to quantify models' reliance on future information beyond cutoff timestamps. Experimental results reveal that LLMs struggle to consistently adhere to temporal cutoffs across common prompting strategies and tasks, demonstrating persistent challenges in ex-ante reasoning. This benchmark provides a potential evaluation framework to advance the development of LLMs' temporal reasoning ability for time-sensitive applications.
Code (1)
Tasks
PredictionQuestion AnsweringStock PredictionSimilar Papers 제목 키워드 기반
InvEvolve: Evolving White-Box Inventory Policies via Large Language Models with Performance Guarantees
We study how large language models can be used to generate inventory policies in online settings with non-stationary demand. Our work is motivated by recent advances in LLM-based evolutionary search, such as AlphaEvolve,…
Reinforcement LearningCost-Optimal LLM Routing with Limited User Feedback under User Satisfaction Guarantees
Inference costs for large language model (LLM) applications are rapidly growing, driven by surging demand and rising infrastructure cost. Users expect high-quality responses, and in commercial settings this is formally c…
Accelerating Masked Diffusion Large Language Models: A Survey of Efficient Inference Techniques
Diffusion large language models (dLLMs) offer a theoretical advantage in parallel generation over standard autoregressive models. However, parallel generation alone does not guarantee practical speedups. Realizing this e…
Efficient Evaluation of LLM Performance with Statistical Guarantees
Exhaustively evaluating many large language models (LLMs) on a large suite of benchmarks is expensive. We cast benchmarking as finite-population inference and, under a fixed query budget, seek tight confidence intervals …
Question SelectionMESS+: Energy-Optimal Inferencing in Language Model Zoos with Service Level Guarantees
Open-weight large language model (LLM) zoos allow users to quickly integrate state-of-the-art models into systems. Despite increasing availability, selecting the most appropriate model for a given task still largely reli…
Language ModelingLanguage ModellingLarge Language ModelModel Selection+1