Memory Architectures for Multi-Turn Text-to-SQL: A Benchmark and Empirical Study

2026-05-25 · Ravi Kumar Tummalapenta, Suman Addanki arxiv

Multi-turn Text-to-SQL is central to enterprise analytics yet remains predominantly evaluated in single-turn settings. We introduce EnterpriseMem-Bench, a multi-turn Text-to-SQL benchmark of 300 sessions and 1,400 turns built programmatically from three enterprise domains (BIRD financial, SEC EDGAR, Northwind), with deterministic ground truth and per-turn memory-critical annotation. We evaluate five frontier models -- GPT-5 mini, GPT-5.2, Claude Sonnet 4.5, Sonnet 4.6, and Opus 4.6 -- across five memory conditions enabling a three-way ablation isolating working-memory window size, episodic retrieval, and semantic augmentation as independent effects. All Claude models are evaluated with extended thinking enabled to maintain parity with GPT reasoning models. We introduce the Memory Benefit Score (MBS) as a per-turn diagnostic metric. Four findings emerge: (1) stateless multi-turn Text-to-SQL collapses to zero execution accuracy by Turn 3 across all five models, even under reasoning; (2) memory-architecture complexity does not monotonically improve accuracy -- working memory dominates, and additional components produce model- and dataset-dependent effects from +14 to -16 percentage points; (3) Claude Sonnet 4.6 underperforms Sonnet 4.5 by 17-33pp on SEC EDGAR across conditions, a generational regression persisting under reasoning; (4) under reasoning, Claude error distributions become mono-modal -- every non-correct turn is a wrong-result error. We release the benchmark, agent, and evaluation code.

📄 PDF Abstract BibTeX arXiv:2605.26394

Code (0)

등록된 구현이 없습니다.

Beyond the Context Window: A Cost-Performance Analysis of Fact-Based Memory vs. Long-Context LLMs for Persistent Agents

2026-03-05 · Natchanon Pollertlam, Witchayut Kornsuwannawit arxiv

Persistent conversational AI systems face a choice between passing full conversation histories to a long-context large language model (LLM) and maintaining a dedicated memory system that extracts and retrieves structured…

MemTool: Optimizing Short-Term Memory Management for Dynamic Tool Calling in LLM Agent Multi-Turn Conversations

2025-07-29 · Elias Lumer, Anmol Gulati, Vamse Kumar Subbiah, Pradeep Honaganahalli Basavaraju 외 arxiv

Large Language Model (LLM) agents have shown significant autonomous capabilities in dynamically searching and incorporating relevant tools or Model Context Protocol (MCP) servers for individual queries. However, fixed co…

CMT-RAG: Complementary Memory Traces for Multi-turn Multi-hop RAG

2026-07-29 · Lang Zhou, Yingjian Chen, Shuxuan Li, Kun-Yu Lin 외 arxiv

Multi-turn information-seeking conversations require both multi-hop reasoning and long-range dependency tracking across turns. However, existing RAG systems typically represent conversational memory as raw dialogue histo…

Beyond Single-Turn: A Survey on Multi-Turn Interactions with Large Language Models

2025-04-07 · Yubo Li, Xiaobin Shen, Xinyu Yao, Xueying Ding 외

Recent advancements in large language models (LLMs) have revolutionized their ability to handle single-turn tasks, yet real-world applications demand sophisticated multi-turn interactions. This survey provides a comprehe…

Dialogue Evaluation Fairness Instruction Following Math

Learning to Retrieve: Dual-Level Long-Term Memory for Text-to-SQL Agents

2026-05-30 · Yibo Wang, Nikki Lijing Kuang, Philip S. Yu, Zhewei Yao 외 arxiv

Interactive text-to-SQL agents solve database tasks through multi-turn interactions involving schema exploration, query execution, feedback interpretation, and decision revision. Long-term memory helps agents reuse past …

Reinforcement Learning