Evaluating the Text-to-SQL Capabilities of Large Language Models
We perform an empirical evaluation of Text-to-SQL capabilities of the Codex language model. We find that, without any finetuning, Codex is a strong baseline on the Spider benchmark; we also analyze the failure modes of Codex in this setting. Furthermore, we demonstrate on the GeoQuery and Scholar benchmarks that a small number of in-domain examples provided in the prompt enables Codex to perform better than state-of-the-art models finetuned on such few-shot examples.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModelingLanguage ModellingText to SQLText-To-SQLSimilar Papers 제목 키워드 기반
MM-Vet v2: A Challenging Benchmark to Evaluate Large Multimodal Models for Integrated Capabilities
MM-Vet, with open-ended vision-language questions targeting at evaluating integrated capabilities, has become one of the most popular benchmarks for large multimodal model evaluation. MM-Vet assesses six core vision-lang…
MathMM-VetMM-Vet v2Optical Character Recognition (OCR)+2Evaluating the Text-to-SQL Capabilities of Large Language Models
We perform an empirical evaluation of Text-to-SQL capabilities of the Codex language model. We find that, without any finetuning, Codex is a strong baseline on the Spider benchmark; we also analyze the failure modes of C…
Language ModelingLanguage ModellingText to SQLText-To-SQLMMR: Evaluating Reading Ability of Large Multimodal Models
Large multimodal models (LMMs) have demonstrated impressive capabilities in understanding various types of image, including text-rich images. Most existing text-rich image benchmarks are simple extraction-based question …
Font RecognitionMMR totalOptical Character Recognition (OCR)Question Answering+2CMMLU: Measuring massive multitask language understanding in Chinese
As the capabilities of large language models (LLMs) continue to advance, evaluating their performance becomes increasingly crucial and challenging. This paper aims to bridge this gap by introducing CMMLU, a comprehensive…
Large Language ModelCLongEval: A Chinese Benchmark for Evaluating Long-Context Large Language Models
Developing Large Language Models (LLMs) with robust long-context capabilities has been the recent research focus, resulting in the emergence of long-context LLMs proficient in Chinese. However, the evaluation of these mo…