If You Think You Can Do Real-World Text-to

Data warehouses are often subject to a preponderance of ad-hoc queries. They came into existence in the 1990s to support historical data stores of customer-facing data, especially sales data. At that time, it was widely reported that they paid for themselves based on better stock rotation in less than six months. For example, if pet rocks are “out” and Barbie dolls are “in,” then discount the pet rocks and move them up front; tie up the manufacturer of Barbie dolls with a big order so competitors cannot get merchandise.

Although there are repeated queries (for example, “produce the weekly sales report”), much of the traffic is ad hoc in nature, including the above example. Historically, data warehouses were fronted by business intelligence tools such as Business Objects, Power BI, and Tableau. This allowed many queries to be coded by business users using a simple graphical interface. Obviously, one would like to query data warehouses and other large, structured data stores in natural language to further empower non-programmers. It would have the added benefit of not requiring familiarity with a bespoke user interface. With the advent of large language models (LLMs) and agentic AI, this becomes a real possibility, and there are public benchmarks with leaderboards that show promising results.

On Spider 1.0,1 the leaderboard had accuracy in the 80+% range, while on Bird-SQL,2 accuracy was above 90%. However, both benchmarks have the following issues:

Spider 2.03 has addressed the last issue, but not the first three. The leaderboard of agentic-AI solutions for this benchmark is in the 90+% range.

In our opinion, real data warehouses have all of the issues mentioned above. Hence, any text-to-SQL benchmark should address the difficulties of real-world data stores. Those that do not are of academic interest only and do not reflect reality.

For the last two years, we have been trying LLM-based text-to-SQL on real workloads from real data warehouses. We started with the above-mentioned MIT data warehouse, an Oracle data warehouse of 1,400+ tables that is queried by many units at MIT (including CSAIL). Typically, business users pose a question to IT personnel who translate the request to SQL, often with clarifying dialog between the two people. We got real queries from the Oracle log and generated the corresponding natural language with the assistance of real users. This generated a collection of (natural language, gold SQL) pairs. Later we applied the same methodology to three other data warehouses. With permission, we anonymized and extended the queries and the data to produce the Beaver benchmark.4 One should clearly note the four issues noted above are all present in this benchmark. As such, we do not expect to get the same high accuracy as Spider/Bird. More details on the benchmark are published online.5

On this benchmark, a pure LLM generated an accuracy score of zero. Adding RAG, prompt engineering, and agentic AI raised accuracy to the 10+% range. See Jindal6 for results on this sort of model. To get a higher score, we gave the LLM the specific tables in the from clause of the gold SQL and the actual join clauses when there were multiple tables. That generated accuracy in the 30+% range. A leaderboard for results on Beaver is available online.7 Note that this is 50+% lower in accuracy that the public benchmarks and is the difference between “the technology has promise” and “the technology does not work.”

We would welcome systems and ideas on how to make LLM/agentic AI technology work much better on real-world business applications, and we have made Beaver publicly available to assist in this process. So have at it! Our own ideas on how to do better are contained in the Rubicon system,8 so take a look.

A simple conclusion can be drawn from this discussion. Good results using traditional LLM/agentic AI technology are likely only on data warehouses with clean schemas, simple queries, and little idiosyncratic data. Otherwise, try Rubicon or your good idea.

1. https://yale-lily.github.io/spider

2. https://bird-bench.github.io/

3. https://spider2-sql.github.io

4. https://beaverbench.github.io/

5. https://beaverbench.github.io/#overview

6. https://arxiv.org/pdf/2602.08320

7. https://beaverbench.github.io/#leaderboard

8. https://arxiv.org/abs/2604.21413

Michael Stonebraker is a professor of Computer Science at the Massachusetts Institute of Technology, where he specializes in database management. For his contributions, he was awarded the ACM A.M. Turing Award in 2014.

Peter Baile Chen is a Ph.D. student at the the Computer Science and Artificial Intelligence Laboratory of the Massachusetts Institute of Technology, where he works with Michael Stonebraker at the intersection of AI and data systems.