Domain Adaptation of a State of the Art Text-to-SQL Model: Lessons Learned and Challenges Found
There are many recent advanced developments for the Text-to-SQL task, where the Picard model is one of the the top performing models as measured by the Spider dataset competition. However, bringing Text-to-SQL systems to realistic use-cases through domain adaptation remains a tough challenge. We analyze how well the base T5 Language Model and Picard perform on query structures different from the Spider dataset, we fine-tuned the base model on the Spider data and on independent databases (DB). To avoid accessing the DB content online during inference, we also present an alternative way to disambiguate the values in an input question using a rule-based approach that relies on an intermediate representation of the semantic concepts of an input question. In our results we show in what cases T5 and Picard can deliver good performance, we share the lessons learned, and discuss current domain adaptation challenges.
Code (0)
등록된 구현이 없습니다.
Tasks
Domain AdaptationLanguage ModelingLanguage ModellingText to SQLText-To-SQLMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Adaptation Odyssey in LLMs: Why Does Additional Pretraining Sometimes Fail to Improve?
In the last decade, the generalization and adaptation abilities of deep learning models were typically evaluated on fixed training and test distributions. Contrary to traditional deep learning, large language models (LLM…
Deep LearningAI-assisted Coding with Cody: Lessons from Context Retrieval and Evaluation for Code Recommendations
In this work, we discuss a recently popular type of recommender system: an LLM-based coding assistant. Connecting the task of providing code recommendations in multiple formats to traditional RecSys challenges, we outlin…
Recommendation SystemsRetrievalDomain adaptation in practice: Lessons from a real-world information extraction pipeline
Advances in transfer learning and domain adaptation have raised hopes that once-challenging NLP tasks are ready to be put to use for sophisticated information extraction needs. In this work, we describe an effort to do j…
Domain AdaptationLink PredictionModel SelectionNegation+3Lessons from Contextual Bandit Learning in a Customer Support Bot
In this work, we describe practical lessons we have learned from successfully using contextual bandits (CBs) to improve key business metrics of the Microsoft Virtual Agent for customer support. While our current use case…
Information RetrievalMulti-Armed BanditsRecommendation SystemsRetrievalCausal Mechanism Transfer Network for Time Series Domain Adaptation in Mechanical Systems
Data-driven models are becoming essential parts in modern mechanical systems, commonly used to capture the behavior of various equipment and varying environmental characteristics. Despite the advantages of these data-dri…
Domain AdaptationFault DetectionTime SeriesTime Series Analysis