Skip to content

Commit 6d6a3ca

Browse files
Morlejclaude
andcommitted
fix(dsql): address remaining PR awslabs#162 review items
- awslabs#17: Downgrade eval results to qualitative comparison, record model and version, note n=1 and recommend n>=3 for production confidence - awslabs#18: SKILL.md is 281 lines (will update PR body) - awslabs#20: Strengthen awsknowledge fallback to MUST — refuse fallback when recommendation depends on exact limit value - awslabs#21: Already addressed in prior commit (reltuples staleness) - awslabs#15: Document manual-only status and future Python converter direction (per anwesham-lab's suggestion for deterministic rewrites) - awslabs#19: MCP mirror PR noted as follow-up in PR body Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent 7f23484 commit 6d6a3ca

2 files changed

Lines changed: 9 additions & 2 deletions

File tree

tools/evals/databases-on-aws/README.md

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -165,10 +165,15 @@ including type coercion index bypass, subquery unnesting, OR-to-IN, GROUP BY pus
165165
DSQL-specific patterns (reltuples estimate, join splitting). Includes one negative case
166166
(OR across different columns — agent should decline).
167167

168-
**Evaluation method:** Manual behavioural comparison. Run `claude -p` with skill loaded vs
168+
**Evaluation method:** Manual qualitative comparison (n=1). Run `claude -p` with skill loaded vs
169169
`claude -p --bare` from a clean directory. Results in `query_plan_rewrite_eval_results.md`.
170170
No automated runner script — this suite is manual-only.
171171

172+
**Future direction:** Many of these rewrites are deterministic pattern transformations. A future
173+
iteration SHOULD implement them as a Python SQL converter script that parses and rewrites SQL
174+
directly, with the reference files serving as documentation for the converter's rules. This
175+
would move correctness-critical rewrites out of the LLM and into deterministic code.
176+
172177
**What it checks** (11 eval prompts):
173178

174179
| Eval | Focus | Key assertions |

tools/evals/databases-on-aws/dsql/query_plan_rewrite_eval_results.md

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,9 @@
11
# Query Plan Rewrite Eval Results — With-Skill vs Baseline
22

33
**Date:** 2026-05-08
4-
**Evaluation method:** Manual behavioral comparison — `claude -p` with skill loaded (from agent-plugins project root) vs `claude -p --bare` from clean directory. PASS/FAIL is a human assessment of transcripts against the expectations in `query_plan_rewrite_evals.json`.
4+
**Model:** Claude Opus 4.6 (global.anthropic.claude-opus-4-6-v1)
5+
**Runs per eval:** 1 (qualitative comparison, not variance-tested)
6+
**Evaluation method:** Manual qualitative comparison — `claude -p` with skill loaded (from agent-plugins project root) vs `claude -p --bare` from clean directory. n=1 per cell; PASS/FAIL is a single human transcript assessment. Results indicate directional improvement, not statistical significance. Re-run with n≥3 and majority vote for production confidence.
57

68
## Summary
79

0 commit comments

Comments
 (0)