Which open models are actually good at SQL, measured the same way
Published figures for text-to-SQL models cannot be compared with each other. Each comes from its authors, under their own prompt and their own way of showing the schema, and the difference that makes is larger than the difference between the models. This page measures them all one way, and publishes the protocol before the run rather than after.
What everyone claims today
These are the figures the authors publish, and the ranking they produce. Read the two columns on the right before the one in the middle. Rows seven and eight are the same models measured differently: XiYanSQL-32B drops five points when the schema is written as DDL instead of M-Schema, and Qwen2.5-Coder-32B is a general model under my own protocol. The leader cannot be used at all, because its weights were never published.
| # | Model | Claimed on BIRD dev | Weights | Source |
|---|---|---|---|---|
| 1 | Arctic-Text2SQL-R1-32B32B · Snowflake | 70.47 | not published | paper |
| 2 | Arctic-Text2SQL-R1-7B7B · Apache-2.0 · Snowflake | 68.9 | downloadable | model card |
| 3 | XiYanSQL-QwenCoder-32B32B · Apache-2.0 · XGenerationLab | 67.14 | downloadable | model card |
| 4 | XiYanSQL-QwenCoder-14B14B · Apache-2.0 · XGenerationLab | 65.32 | downloadable | model card |
| 5 | OmniSQL-32B32B · Apache-2.0 · RUC | 64.5 | downloadable | model card |
| 6 | XiYanSQL-QwenCoder-7B7B · Apache-2.0 · XGenerationLab | 62.13 | downloadable | model card |
| 7 | XiYanSQL-QwenCoder-32B32B, the same model, schema written as DDL | 62.26 | downloadable | model card |
| 8 | Qwen2.5-Coder-32B32B · Apache-2.0 · general purpose, with self-correction | 51.63 | downloadable | my measurement |
Authors' own figures, each under its own conditions. Not a like-for-like comparison.
What that table does not tell you
Nothing in it is comparable. Each number comes from its authors under their own prompt, their own schema format and their own decoding, on a benchmark whose annotations are wrong in half the cases. A five point swing from the schema format alone is larger than the gap between most of these rows. That is the entire reason for the measurement below.
Results
The run has not happened yet. This table is filled by the build from the results files, and the build refuses to publish a figure that no results file backs. Until a run exists, there is nothing here, and that is deliberate rather than an omission.
Participants in the first wave: Arctic-Text2SQL-R1-7B, XiYanSQL-QwenCoder at 7B, 14B and 32B, OmniSQL-32B, Qwen2.5-Coder at 7B, 14B and 32B as the baseline that ties these numbers to arXiv:2606.29733, and Qwen3-Coder-30B-A3B. Arctic-Text2SQL-R1-32B is excluded: its weights are not published, so its reported figure can be neither reproduced nor deployed.
The protocol, fixed before the run
Everything here was written down before any model was measured, and it is in the repository with the commit that fixed it. If any of it changes, the run is repeated, not edited.
- The same questions for every participant, in the same order, from a deterministic sample.
- One prompt, greedy decoding, a single call, no examples in the context, unless the thing being measured is one of those.
- The same context budget and the same execution timeout for everyone.
- Our own engine runs through the same interface as everyone else, with no privileges.
- Per-question predictions are published for every run, including the wrong ones.
- Confidence intervals by bootstrap over questions, 2000 resamples, seed 0. A verdict that moves between seeds 0 and 5 is recorded as undetermined.
What is measured, and on what
| Dataset | What it is | Size | What it settles |
|---|---|---|---|
| BIRD dev | the public English standard | 1534 | comparability with the published literature |
| BIRD dev, cleaned | the same set after the authors' quality pass | 1534 | how much of our own result is annotation noise |
| RedSQL | public Russian, domain-specific | 409 | an outside reference point in a second language |
| Our held-out set | frozen with a SHA-256 digest before any tool sees it | 600 | the verdict, and the gap between languages |
The held-out set is one set of schemas and one gold query per task, with four wordings of the question: two English, two Russian. That is what makes the gap between languages measurable on an identical target, which no published set offers. The Russian wordings are written from the meaning, never translated from the English, or the measurement collapses into a test of translation quality.
Why a set of our own, when public benchmarks exist. A CIDR 2026 study by Jin, Choi, Zhu and Kang found annotation errors in 52.8% of BIRD Mini-Dev problems and 66.1% of the Spider 2.0-Snow problems with open gold queries. Correcting them moved leading systems by up to 19 points and by as much as three places. A score on a public benchmark is a comparison with other papers, not a promise about anyone's data.
Thresholds and the verdict rule
These were fixed before the run as well, so that a result cannot be reinterpreted once it is known.
| Threshold | Value |
|---|---|
| The recipe does no harm | our engine on the held-out set is not below the best available single model under the same protocol |
| English, public set | not below 65% on BIRD dev |
| English, our set | not below the best closed model under the same protocol on the same set |
| The second language works | Russian no more than 10 points behind English |
| Safety | 100% of attempts to go beyond SELECT blocked, 0 queries executed without a row limit and a timeout |
One verdict rule for all of them. Met, if the whole 95% interval lies on the right side of the threshold. Not met, if the whole interval lies on the wrong side. Otherwise undetermined, and it is reported as undetermined rather than rounded into an answer.
What this page will not do
It will not quote a figure from someone else's model card next to our own measurements as though they were the same kind of thing. Those numbers appear in prose, with a link to their source, and never in this table. It will not drop a model because the result was inconvenient: the participant list above is the list that gets published.
Every failure is reviewed by hand before a verdict, at least twenty random ones per run, and split into errors of the model and false alarms of the evaluator. The share of false alarms is published next to the accuracy. That check exists because an evaluator can be wrong in exactly the way a benchmark's annotations can.