DEMO: SQL Diagram Automation with PNG Previews#25
Conversation
This commit introduces an automated pipeline for SQL diagram generation that simplifies the workflow for technical writers. Changes: - Add CI workflow (.github/workflows/docgen-diagrams-ci.yml) that: - Triggers on changes to diagrams.go, sql.y, or the docgen-diagram label - Generates BNF files from sql.y - Builds SVG diagrams via Railroad Diagram Generator - Runs comprehensive validation tests - Packages and uploads artifacts - Triggers bot to sync diagrams to generated-diagrams repo - Comments on PRs with validation status - Add validation test suite (pkg/cmd/docgen/tests/): - TestMissingBNFFiles: Detects missing BNF files - TestMissingHTMLDiagrams: Detects missing HTML output - TestStmtNamingConflicts: Detects _stmt naming conflicts - TestReplaceUnlinkMismatch: Warns about potential broken links - TestBrokenGrammarLinks: Finds invalid grammar references - TestUnusedDiagrams: Reports diagrams not used in docs - TestSKIPDOCSuppressions: Reports SKIP DOC annotations - Add bot sync script (build/scripts/sync-diagrams-to-generated-repo.sh): - Clones generated-diagrams repository - Updates BNF and SVG files - Creates PR with validation summary (no auto-merge) - Assigns PR to docs-infra-prs team - Add writer documentation (docs/generated/sql/bnf/README.md): - Instructions for adding/modifying diagrams - Explains what writers don't need to do (no Bazel commands) - Troubleshooting guide - Add technical documentation (docs/tech-notes/sql-diagram-automation.md): - Architecture overview with diagram - CI workflow details - Bot service specification - Secret configuration - Troubleshooting and rollback procedures New writer workflow: 1. Edit pkg/cmd/docgen/diagrams.go only 2. Open PR with label docgen-diagram 3. Add remote_include manually only for new diagrams Release Notes: None Epic: None
…t repo This commit improves the SQL diagram automation CI workflow: 1. Added changed_files output to track which diagrams changed 2. Updated manifest.json to include changed files as a JSON array 3. Made target repository configurable via DIAGRAM_TARGET_REPO variable (defaults to cockroachdb/generated-diagrams) 4. Enhanced PR comment to show list of changed diagrams 5. Updated TestUnusedDiagrams stub to clarify it's a placeholder for future functionality Epic: None Release note: None
This commit removes the pull_request_target event trigger from the SQL diagram generation workflow to address a security vulnerability. The issue was that pull_request_target grants access to secrets and runs with write permissions. When combined with checking out untrusted PR code and executing scripts from it (./dev, ./build/github/*.sh), this created a vector for secret exfiltration and runner compromise. Changes: - Remove pull_request_target trigger (used for docgen-diagram label) - Remove label check from should_run logic - Add comment explaining the security rationale - workflow_dispatch remains available for manual triggering by maintainers The pull_request trigger (for path changes) remains safe since it does not have access to secrets. Epic: None Release note: None
docgen: Implement complete SQL diagram automation pipeline
This test change adds 'opt_collate' to the inline list for the add_column diagram to demonstrate the SQL diagram automation workflow. Epic: None Release note: None
This commit is FORK-SPECIFIC and should NOT be upstreamed. Changes for running on forks without CockroachDB infrastructure: - Use ubuntu-latest instead of self-hosted runners - Add Bazel installation - Skip EngFlow keys (internal to CockroachDB) Epic: None Release note: None
Testing the automated SQL diagram generation workflow with GitHub-hosted runners. Epic: None Release note: None
- Add Java 11 setup (required for Railroad JAR) - Download and configure Railroad Diagram Generator JAR - Set COCKROACH_REQUIRE_RAILROAD environment variable - Add fallback build command with explicit JAR location This should enable HTML/SVG diagram generation on GitHub-hosted runners. Epic: None Release note: None
The unzip was conflicting with existing LICENSE file in the repo. Now extracting in a temporary directory to avoid conflicts. Epic: None Release note: None
Adding diagnostic output to understand why HTML files aren't being generated. Epic: None Release note: None
Testing the workflow with debug output to see why HTML files aren't generated. Epic: None Release note: None
The dev wrapper requires initialization which doesn't work well on GitHub-hosted runners. Using bazel directly for: - BNF generation - SVG diagram building - Validation tests This should fix the HTML generation issue. Epic: None Release note: None
Previously, writers had no way to see rendered railroad diagrams directly in the PR. They had to either download artifacts, run locally, or wait for the generated-diagrams PR to be merged. This change modifies the docgen-diagrams-ci workflow to embed diagram SVGs directly in the PR comment with inline CSS styling. The workflow now: - Downloads diagram artifacts in the comment-on-pr job - Extracts SVG content from changed HTML files - Adds inline CSS for proper rendering (terminal/nonterminal boxes, lines) - Embeds up to 5 diagrams (under 40KB each) in the PR comment - Shows a note for large diagrams and remaining diagrams Release note: None Epic: None Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
SQL Diagram Generation Report✅ Validation Passed ✔️ No diagram changes detected SKIP DOC WarningsThe following grammar rules are suppressed from documentation: This comment was generated by the SQL Diagram CI workflow. |
Test: SVG Diagram RenderingTesting if GitHub renders inline SVG with CSS styling: <style>.terminal{fill:#FFFFCC;stroke:#000;stroke-width:1}.nonterminal{fill:#FFCCCC;stroke:#000;stroke-width:1}.line{stroke:#000;stroke-width:1;fill:none}text{font-family:monospace;font-size:12px;fill:#000}polygon{fill:#000}</style>
ABORT
opt_abort_mod
If you see a railroad diagram with yellow and pink boxes above, SVG rendering works! |
GitHub sanitizes inline SVG in comments, so the previous approach of embedding SVG directly didn't work. This change switches to PNG images: - Added Setup Node.js step in generate-diagrams job - Added Generate PNG previews step using sharp library (300 DPI) - Updated sync-to-generated-diagrams to copy PNGs to grammar_png/ - Rewrote comment-on-pr to use image markdown with raw.githubusercontent.com URLs The PNG images are pushed to the generated-diagrams repo and referenced via raw.githubusercontent.com URLs, which GitHub renders correctly. Release note: None Epic: None
This minor change triggers the CI workflow to test the PNG preview feature in PR comments. Release note: None Epic: None
SQL Diagram Generation Report✔️ No diagram changes detected This comment was generated by the SQL Diagram CI workflow. |
SQL Diagram Generation Report✅ Validation Passed ✔️ No diagram changes detected SKIP DOC WarningsThe following grammar rules are suppressed from documentation: This comment was generated by the SQL Diagram CI workflow. |
1 similar comment
SQL Diagram Generation Report✅ Validation Passed ✔️ No diagram changes detected SKIP DOC WarningsThe following grammar rules are suppressed from documentation: This comment was generated by the SQL Diagram CI workflow. |
Add the 'region' element to the unlink list for the alter_database_drop_region diagram to test the PNG preview feature. Release note: None Epic: None
2826d21 to
d927e38
Compare
The previous change detection only looked at BNF file differences. This update also checks if diagrams.go was modified compared to the base branch, which catches rendering changes like unlink, replace, or nosplit that don't affect BNF output but do change the diagrams. Release note: None Epic: None
SQL Diagram Generation Report✔️ No diagram changes detected This comment was generated by the SQL Diagram CI workflow. |
SQL Diagram Generation Report✅ Validation Passed 🔄 Diagram changes detected SKIP DOC WarningsThe following grammar rules are suppressed from documentation: This comment was generated by the SQL Diagram CI workflow. |
The script was created in /tmp but node_modules was installed in the workspace directory. Changed to create the script in the workspace where the require() call can find the sharp module. Release note: None Epic: None
7d02d79 to
5ea9dd7
Compare
The heredoc content starting at column 1 was breaking YAML parsing. Replaced with node -e inline script which keeps everything in the YAML string context. Release note: None Epic: None
SQL Diagram Generation Report✅ Validation Passed 🔄 Diagram changes detected SKIP DOC WarningsThe following grammar rules are suppressed from documentation: This comment was generated by the SQL Diagram CI workflow. |
The HTML files are generated in _bazel/bin/ not bazel-bin/. Updated all references to use the correct path. Release note: None Epic: None
SQL Diagram Generation Report✅ Validation Passed 🔄 Diagram changes detected Diagram Previewsabort add_column add_constraint alter alter_backup Showing 5 diagram preview(s). SKIP DOC WarningsThe following grammar rules are suppressed from documentation: This comment was generated by the SQL Diagram CI workflow. |
Use GitHub's <details> tag to hide the verbose SKIP DOC warnings by default, reducing clutter in PR comments. Release note: None Epic: None
SQL Diagram Generation Report✅ Validation Passed 🔄 Diagram changes detected Diagram Previewsabort add_column add_constraint alter alter_backup Showing 5 diagram preview(s). SKIP DOC Warnings (click to expand)The following grammar rules are suppressed from documentation: This comment was generated by the SQL Diagram CI workflow. |










Demo: SQL Diagram Automation Pipeline
This PR demonstrates the complete SQL diagram automation system for CockroachDB documentation.
What This PR Shows
diagrams.gogenerated-diagramsrepo via PRHow It Works
pkg/cmd/docgen/diagrams.goSee the PR Comment Below
The workflow posted a comment showing:
Writer Workflow (New)
Writers only need to:
pkg/cmd/docgen/diagrams.goremote_includeonly for new diagramsNo Bazel commands. No local generation. No engineering coordination required.