A System and Benchmark for LLM-based Q&A on Heterogeneous Data
In many industrial settings, users wish to ask questions whose answers may be found in structured data sources such as a spreadsheets, databases, APIs, or combinations thereof. Often, the user doesn't know how to identify or access the right data source. This problem is compounded even further if multiple (and potentially siloed) data sources must be assembled to derive the answer. Recently, various Text-to-SQL applications that leverage Large Language Models (LLMs) have addressed some of these problems by enabling users to ask questions in natural language. However, these applications remain impractical in realistic industrial settings because they fail to cope with the data source heterogeneity that typifies such environments. In this paper, we address heterogeneity by introducing the siwarex platform, which enables seamless natural language access to both databases and APIs. To demonstrate the effectiveness of siwarex, we extend the popular Spider dataset and benchmark by replacing some of its tables by data retrieval APIs. We find that siwarex does a good job of coping with data source heterogeneity. Our modified Spider benchmark will soon be available to the research community
Code (0)
등록된 구현이 없습니다.
Tasks
Text to SQLText-To-SQLSimilar Papers 제목 키워드 기반
Heterogeneous Federated Learning System for Sparse Healthcare Time-Series Prediction
In this paper, we propose a heterogeneous federated learning (HFL) system for sparse time series prediction in healthcare, which is a decentralized federated learning algorithm with heterogeneous transfers. We design den…
Federated LearningPredictionTime SeriesTime Series Prediction+1CompMix: A Benchmark for Heterogeneous Question Answering
Fact-centric question answering (QA) often requires access to multiple, heterogeneous, information sources. By jointly considering several sources like a knowledge base (KB), a text collection, and tables from the web, Q…
Question AnsweringHTPS: Heterogeneous Transferring Prediction System for Healthcare Datasets
Medical internet of things leads to revolutionary im- provements in medical services, also known as smart healthcare. With the big healthcare data, data mining and machine learning can assist wellness management and inte…
Feature EngineeringManagementPredictionCITE: A Comprehensive Benchmark for Heterogeneous Text-Attributed Graphs on Catalytic Materials
Text-attributed graphs(TAGs) are pervasive in real-world systems,where each node carries its own textual features. In many cases these graphs are inherently heterogeneous, containing multiple node types and diverse edge …
Representation LearningNode ClassificationWatt Counts: Energy-Aware Benchmark for Sustainable LLM Inference on Heterogeneous GPU Architectures
While the large energy consumption of Large Language Models (LLMs) is recognized by the community, system operators lack guidance for energy-efficient LLM inference deployments that leverage energy trade-offs of heteroge…