Skip to content

Commit 0b66f96

Browse files
docs: fix BIRD score framing and MySQL dialect wording
- Remove specific "60-70%" BIRD claim from intro to avoid contradiction with the 41.80%/38.25% direct-generation results shown later (those higher figures come from specialized systems with schema linking) - Reword MySQL "forbids" to "prompts exclude" -- REGEXP_REPLACE and CONVERT_TZ are valid MySQL functions; the pipeline excluded them for portability, not because the dialect forbids them
1 parent 5412153 commit 0b66f96

1 file changed

Lines changed: 2 additions & 2 deletions

File tree

docs/devnotes/posts/text-to-sql.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@ While LLMs have mastered generic coding, Text-to-SQL remains one of the most cha
2020

2121
## **The "Real-World" Gap: Why Academic Data Wasn't Enough**
2222

23-
The gap between academic benchmarks and the messy reality of enterprise data warehouses is massive. On academic benchmarks like Spider (where schemas are clean, tables are few, and queries are straightforward), frontier models score above 85%. On [BIRD](https://bird-bench.github.io/) (which introduces dirty data, larger schemas, and external knowledge requirements), the same models drop to 60-70%. On [Spider 2.0 Lite](https://spider2-sql.github.io/) (which uses real enterprise databases with hundreds of tables, multiple dialects, and complex business logic), even the best models score below 50%.
23+
The gap between academic benchmarks and the messy reality of enterprise data warehouses is massive. On academic benchmarks like Spider (where schemas are clean, tables are few, and queries are straightforward), frontier models score above 85%. On [BIRD](https://bird-bench.github.io/) (which introduces dirty data, larger schemas, and external knowledge requirements), performance drops significantly --- and on [Spider 2.0 Lite](https://spider2-sql.github.io/) (which uses real enterprise databases with hundreds of tables, multiple dialects, and complex business logic), even the best models score below 50%.
2424

2525
The problem isn't model capability --- it's **training data**. Most open-source text-to-SQL datasets assume a "happy path": intuitive column names, perfect data types, and straightforward questions. Production SQL is different:
2626

@@ -307,7 +307,7 @@ The SQL generation step receives the natural-language prompt and the generated s
307307
- **Handle dirty data** -- the query must clean data issues (CAST, REPLACE, SUBSTR, regex) before computing results
308308
- **Ignore distractors** -- no unnecessary joins or column selections; distractor elements must be left untouched
309309
- **Anchor relative time** -- instead of `CURRENT_DATE`, anchor to `(SELECT MAX(date_col) FROM table)` for reproducibility
310-
- **Dialect-specific syntax** -- SQLite uses `strftime`, MySQL uses `DATE_SUB`, PostgreSQL uses `::` casting and `interval`. Each dialect also has its own explicit limitations (e.g., SQLite forbids `LATERAL` joins and `REGEXP_REPLACE`; MySQL forbids `REGEXP_REPLACE` and `CONVERT_TZ`)
310+
- **Dialect-specific syntax** -- SQLite uses `strftime`, MySQL uses `DATE_SUB`, PostgreSQL uses `::` casting and `interval`. Each dialect also has prompt-level constraints to ensure portability (e.g., SQLite prompts exclude `LATERAL` joins and `REGEXP_REPLACE`; MySQL prompts exclude `REGEXP_REPLACE` for pre-8.0 compatibility and `CONVERT_TZ` to avoid unpopulated timezone table issues)
311311

312312
```python
313313
config.add_column(dd.LLMCodeColumnConfig(

0 commit comments

Comments
 (0)