AlloBench: Measuring Online Tool Allocation Capability in LLM Agents
Creating a reusable tool is an investment: an agent pays a fixed cost now in exchange for the potential of future reuse. Therefore, a user should prefer an agent that creates a small number of highly reusable tools, rather than many one-offs. We introduce a paired benchmark that tests whether LLM agents exhibit conscious allocation behavior under a fixed budget in two contexts: an abstract text-based formulation and a code-construction task. We find that every frontier model we test---Claude Haiku, Claude Opus, GPT-5.4-mini, and GPT-5.6 Sol---acts near-optimally in the abstract framing but fails to transfer this ability to script-writing. Through further experiments, we identify the particular failure modes for each model. Notably, the first three models fail even when the scripts are not evaluated, while GPT-5.6 Sol stays selective under that weaker manipulation and collapses only at full construction. Furthermore, an open-source Qwen model policy-trained for abstract allocation generalizes this ability across held-out lexical variations, but sees no improvement at script allocation. Together, these results establish online tool allocation as a significant capability boundary, even for modern frontier models.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Spending Money Wisely: Online Electronic Coupon Allocation based on Real-Time User Intent Detection
Online electronic coupon (e-coupon) is becoming a primary tool for e-commerce platforms to attract users to place orders. E-coupons are the digital equivalent of traditional paper coupons which provide customers with dis…
Intent DetectionMultiple-choiceA new neural-network-based model for measuring the strength of a pseudorandom binary sequence
Maximum order complexity is an important tool for measuring the nonlinearity of a pseudorandom sequence. There is a lack of tools for predicting the strength of a pseudorandom binary sequence in an effective and efficien…
Evaluating Human-AI Safety: A Framework for Measuring Harmful Capability Uplift
Current frontier AI safety evaluations emphasize static benchmarks, third-party annotations, and red-teaming. In this position paper, we argue that AI safety research should focus on human-centered evaluations that measu…
A Unified Framework for Marketing Budget Allocation
While marketing budget allocation has been studied for decades in traditional business, nowadays online business brings much more challenges due to the dynamic environment and complex decision-making process. In this pap…
Decision MakingMarketingAchievement and Fragility of Long-term Equitability
Equipping current decision-making tools with notions of fairness, equitability, or other ethically motivated outcomes, is one of the top priorities in recent research efforts in machine learning, AI, and optimization. In…
Decision MakingFairness