Mind the Data Gap: Bridging LLMs to Enterprise Data Integration
Leading large language models (LLMs) are trained on public data. However, most of the world's data is dark data that is not publicly accessible, mainly in the form of private organizational or enterprise data. We show that the performance of methods based on LLMs seriously degrades when tested on real-world enterprise datasets. Current benchmarks, based on public data, overestimate the performance of LLMs. We release a new benchmark dataset, the GOBY Benchmark, to advance discovery in enterprise data integration. Based on our experience with this enterprise benchmark, we propose techniques to uplift the performance of LLMs on enterprise data, including (1) hierarchical annotation, (2) runtime class-learning, and (3) ontology synthesis. We show that, once these techniques are deployed, the performance on enterprise data becomes on par with that of public data. The Goby benchmark can be obtained at https://goby-benchmark.github.io/.
Code (0)
등록된 구현이 없습니다.
Tasks
Data IntegrationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Red Teaming the Mind of the Machine: A Systematic Evaluation of Prompt Injection and Jailbreak Vulnerabilities in LLMs
Large Language Models (LLMs) are increasingly integrated into consumer and enterprise applications. Despite their capabilities, they remain susceptible to adversarial attacks such as prompt injection and jailbreaks that …
Red TeamingMindfulDiary: Harnessing Large Language Model to Support Psychiatric Patients' Journaling
In the mental health domain, Large Language Models (LLMs) offer promising new opportunities, though their inherent complexity and low controllability have raised questions about their suitability in clinical settings. We…
Language ModelingLanguage ModellingLarge Language ModelSheetMind: An End-to-End LLM-Powered Multi-Agent Framework for Spreadsheet Automation
We present SheetMind, a modular multi-agent framework powered by large language models (LLMs) for spreadsheet automation via natural language instructions. The system comprises three specialized agents: a Manager Agent t…
Making LLMs Work for Enterprise Data Tasks
Large language models (LLMs) know little about enterprise database tables in the private data ecosystem, which substantially differ from web text in structure and content. As LLMs' performance is tied to their training d…
ManagementText to SQLText-To-SQLVLMs-in-the-Wild: Bridging the Gap Between Academic Benchmarks and Enterprise Reality
Open-source Vision-Language Models show immense promise for enterprise applications, yet a critical disconnect exists between academic evaluation and enterprise deployment requirements. Current benchmarks rely heavily on…
Object Detection