ComplexFuncBench: Exploring Multi-Step and Constrained Function Calling under Long-Context Scenario
Enhancing large language models (LLMs) with real-time APIs can help generate more accurate and up-to-date responses. However, evaluating the function calling abilities of LLMs in real-world scenarios remains under-explored due to the complexity of data collection and evaluation. In this work, we introduce ComplexFuncBench, a benchmark for complex function calling across five real-world scenarios. Compared to existing benchmarks, ComplexFuncBench encompasses multi-step and constrained function calling, which requires long-parameter filing, parameter value reasoning, and 128k long context. Additionally, we propose an automatic framework, ComplexEval, for quantitatively evaluating complex function calling tasks. Through comprehensive experiments, we demonstrate the deficiencies of state-of-the-art LLMs in function calling and suggest future directions for optimizing these capabilities. The data and code are available at \url{https://github.com/THUDM/ComplexFuncBench}.
Code (1)
Similar Papers 제목 키워드 기반
Training LLMs for Multi-Step Tool Orchestration with Constrained Data Synthesis and Graduated Rewards
Multi-step tool orchestration remains challenging for LLMs, as state-of-the-art models frequently fail on full sequence execution due to parameter errors. Training for these workflows faces two obstacles: the lack of env…
Reinforcement LearningConstrained Bayesian Optimization under Bivariate Gaussian Process with Application to Cure Process Optimization
Bayesian Optimization, leveraging Gaussian process models, has proven to be a powerful tool for minimizing expensive-to-evaluate objective functions by efficiently exploring the search space. Extensions such as constrain…
Bayesian OptimizationGaussian ProcessesConstrained Two-step Look-Ahead Bayesian Optimization
Recent advances in computationally efficient non-myopic Bayesian optimization offer improved query efficiency over traditional myopic methods like expected improvement, with only a modest increase in computational cost. …
Bayesian OptimizationVocal Bursts Valence PredictionReLU Strikes Back: Exploiting Activation Sparsity in Large Language Models
Large Language Models (LLMs) with billions of parameters have drastically transformed AI applications. However, their demanding computation during inference has raised significant challenges for deployment on resource-co…
A Constrained BA Algorithm for Rate-Distortion and Distortion-Rate Functions
The Blahut-Arimoto (BA) algorithm has played a fundamental role in the numerical computation of rate-distortion (RD) functions. This algorithm possesses a desirable monotonic convergence property by alternatively minimiz…