Skip to content

Commit 1a237d9

Browse files
fix: text-to-sql devnote date, images, and publish-devnotes nav (#546)
- Update post date from 2026-03-11 to 2026-04-14 so it appears as the newest post on the devnotes page. - Replace raw <img> tags with markdown image syntax so mkdocs rewrites relative paths correctly for the blog plugin's slug-based URLs. - Overlay mkdocs.yml from HEAD in publish-devnotes workflow so new nav entries are included in devnotes-only rebuilds.
1 parent 1448f9c commit 1a237d9

2 files changed

Lines changed: 13 additions & 11 deletions

File tree

.github/workflows/publish-devnotes.yml

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,15 @@ jobs:
3939
run: |
4040
git checkout ${{ env.SOURCE_SHA }}
4141
git checkout ${{ github.sha }} -- docs/devnotes/
42+
43+
# Patch the "Dev Notes" nav section from HEAD's mkdocs.yml into the
44+
# old source's mkdocs.yml. This keeps nav entries for new devnotes
45+
# without pulling in entries for non-devnotes pages that may not
46+
# exist in the old source checkout.
47+
pip install -q yq
48+
DEVNOTES_NAV=$(git show ${{ github.sha }}:mkdocs.yml \
49+
| yq '.nav[] | select(has("Dev Notes")) | .["Dev Notes"]')
50+
yq -y -i "(.nav[] | select(has(\"Dev Notes\")))[\"Dev Notes\"] = ${DEVNOTES_NAV}" mkdocs.yml
4251
- name: Install uv
4352
uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
4453
with:

docs/devnotes/posts/text-to-sql.md

Lines changed: 4 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
---
2-
date: 2026-03-11
2+
date: 2026-04-14
33
authors:
44
- dnathawani
55
- ymeyer
@@ -8,9 +8,7 @@ authors:
88

99
# **Engineering an Enterprise-Grade Text-to-SQL Dataset with NeMo Data Designer**
1010

11-
<img src="assets/text-to-sql/text-to-sql-pipeline.jpg" alt="Text-to-SQL Synthetic Data Pipeline" width="800">
12-
13-
<br>
11+
![Text-to-SQL Synthetic Data Pipeline](assets/text-to-sql/text-to-sql-pipeline.jpg){ width=800 }
1412

1513
While LLMs have mastered generic coding, Text-to-SQL remains one of the most challenging frontiers in enterprise AI. In many ways this is due to (i) SQL tasks relying on both code and data and (ii) real-world data and databases being quite messy. Focusing on careful data design that accounts for real-world diversity and complexity, we built a [NeMo Data Designer](https://github.com/NVIDIA-NeMo/DataDesigner) pipeline that includes conditional sampling, three-stage LLM generation, code validators, and multi-dimensional judge scoring to generate reasoning-heavy text-to-SQL samples across PostgreSQL, MySQL, and SQLite, and automatically filter down to the highest quality 96.5k records. Each sample pairs a natural-language prompt and a fully synthetic database schema context with a target SQL query. To improve robustness and mimic the messiness of production databases, the pipeline injects distractor tables and columns into the schema context, forcing the model to learn to ignore irrelevant schema elements. The final dataset is validated and filtered through per-dialect syntax validators and five LLM-as-a-critic judges.
1614

@@ -426,9 +424,7 @@ The high rejection rate is a feature, not a bug. By generating 3x more data than
426424

427425
This dataset was shipped in the SFT stage of **Nemotron Super v3**. On the [BIRD SQL benchmark](https://bird-bench.github.io/) (1,534 dev samples, 5-run average), Nemotron Super achieves **41.80% EX** (execution accuracy) --- outperforming GPT-OSS-120B at 38.25%. Including our synthetic dataset in the SFT blend raised Nemotron Super's EX on BIRD by **15 points**, from 26.77% to 41.80%.
428426

429-
<img src="assets/text-to-sql/bird-benchmark-results.jpg" alt="BIRD SQL Benchmark Results — Nemotron Super EX improves from 26.77% to 41.80%" width="800">
430-
431-
<br>
427+
![BIRD SQL Benchmark Results - Nemotron Super EX improves from 26.77% to 41.80%](assets/text-to-sql/bird-benchmark-results.jpg){ width=800 }
432428

433429
| Model | BIRD EX (%) |
434430
|-------|-------------|
@@ -465,7 +461,7 @@ This dataset was shipped in the SFT stage of **Nemotron Super v3**. On the [BIRD
465461
- **Code Sandbox for semantic correctness.** The current Quality Waterfall validates syntax and assesses quality (LLM judges), but it doesn't verify whether the query actually returns the right results. A natural next step would be adding Code Sandbox support to Data Designer --- executing generated SQL against a ground-truth database and comparing results to enable execution-based filtering, end-to-end verification, and hard negative mining for preference training.
466462
- **RL on BIRD.** Run reinforcement learning experiments using the [NeMo Gym](https://github.com/NVIDIA-NeMo/Gym) RL environment for BIRD, training models to improve execution accuracy through reward signals from actual query execution.
467463
- **Schema representation.** Improve how schemas are represented in prompts to close the gap with SOTA approaches that use richer structural encodings (e.g., foreign key graphs, column descriptions, value examples).
468-
- **More benchmarks.** Incorporate additional SQL benchmarks --- [Spider 2.0](https://spider2-sql.github.io/), [LiveSQLBench](https://livesqlbench.github.io/) --- to evaluate generalization beyond BIRD and drive the next iteration of the pipeline.
464+
- **More benchmarks.** Incorporate additional SQL benchmarks --- [Spider 2.0](https://spider2-sql.github.io/), [LiveSQLBench](https://livesqlbench.ai/) --- to evaluate generalization beyond BIRD and drive the next iteration of the pipeline.
469465

470466
---
471467

@@ -590,9 +586,6 @@ Because this pipeline is encapsulated in Data Designer, the configuration can be
590586
- **NeMo Data Designer:** [github.com/NVIDIA-NeMo/DataDesigner](https://github.com/NVIDIA-NeMo/DataDesigner)
591587
- **BIRD Benchmark:** [bird-bench.github.io](https://bird-bench.github.io/)
592588
- **Spider 2.0 Benchmark:** [spider2-sql.github.io](https://spider2-sql.github.io/)
593-
- **Structured Outputs Dev Note** (related pipeline): [Structured Outputs for Nemotron](structured-outputs-from-nemotron.md)
594-
- **RQA Dev Note** (reasoning data with Data Designer): [Graduate-Level Science Reasoning Data](rqa.md)
595-
596589
---
597590

598591
*Want to learn more about NeMo Data Designer? Check out our [documentation](https://github.com/NVIDIA-NeMo/DataDesigner) and start building your own high-fidelity synthetic datasets today.*

0 commit comments

Comments
 (0)