Skip to content

Commit a163de9

Browse files
Morlejclaude
andcommitted
fix(dsql): bump databases-on-aws to 1.6.0, add Workflow 12 evals
Resolves the 1.5.0 version collision with EF Core PR awslabs#208 (both this branch's system-diagnostics work and awslabs#208 independently claimed 1.5.0). Bumps to 1.6.0 across plugin.json, codex plugin.json, and marketplace.json. Adds eval coverage for Workflow 12 (system diagnostics), which previously had none: 3 triggering cases and 1 functional AAS-interpretation eval. Corrects stale eval README counts (37 trigger cases, 15 functional prompts / 54 assertions) and documents evals 13-15. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent bb80829 commit a163de9

6 files changed

Lines changed: 51 additions & 22 deletions

File tree

.claude-plugin/marketplace.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -164,7 +164,7 @@
164164
"name": "databases-on-aws",
165165
"source": "./plugins/databases-on-aws",
166166
"tags": ["aws", "database", "aurora", "dsql", "serverless", "postgresql"],
167-
"version": "1.5.0"
167+
"version": "1.6.0"
168168
},
169169
{
170170
"category": "deployment",

plugins/databases-on-aws/.claude-plugin/plugin.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,5 +22,5 @@
2222
"license": "Apache-2.0",
2323
"name": "databases-on-aws",
2424
"repository": "https://github.com/awslabs/agent-plugins",
25-
"version": "1.5.0"
25+
"version": "1.6.0"
2626
}

plugins/databases-on-aws/.codex-plugin/plugin.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
{
22
"name": "databases-on-aws",
3-
"version": "1.5.0",
3+
"version": "1.6.0",
44
"description": "Expert database guidance for the AWS database portfolio. Design schemas, execute queries, handle migrations, and choose the right database for your workload.",
55
"author": {
66
"name": "Amazon Web Services",

tools/evals/databases-on-aws/README.md

Lines changed: 22 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -14,8 +14,8 @@ scripts, and unit tests for that database's skill.
1414
tools/evals/databases-on-aws/
1515
├── README.md # This file — top-level index
1616
└── dsql/ # Aurora DSQL skill evals
17-
├── evals.json # Tier 2: functional evals (9 prompts, 31 assertions)
18-
├── trigger_evals.json # Tier 1: triggering evals (31 test cases)
17+
├── evals.json # Tier 2: functional evals (15 prompts, 54 assertions)
18+
├── trigger_evals.json # Tier 1: triggering evals (37 test cases)
1919
├── safe_query_evals.json # Tier 3: safe_query enforcement (6 prompts, ~30 expectations)
2020
├── query_explainability_evals.json # Workflow 9: query plan diagnostics (9 prompts, 70 assertions)
2121
├── query_plan_rewrite_evals.json # Query rewrites: type coercion, subquery unnesting, etc. (11 prompts, manual)
@@ -55,7 +55,7 @@ PYTHONPATH="<skill-creator-path>:$PYTHONPATH" python -m scripts.run_eval \
5555

5656
**What it checks:**
5757

58-
- 16 should-trigger prompts (Aurora DSQL, distributed SQL, DSQL migrations, query plan explainability, data loading, etc.)
58+
- 22 should-trigger prompts (Aurora DSQL, distributed SQL, DSQL migrations, query plan explainability, system diagnostics / cluster performance, data loading, etc.)
5959
- 15 should-not-trigger prompts (DynamoDB, Aurora/RDS PostgreSQL with EXPLAIN ANALYZE, Redshift, generic SQL, non-DSQL bulk loading, etc.)
6060

6161
### Tier 2: Functional Evals
@@ -81,22 +81,25 @@ python tools/evals/databases-on-aws/dsql/scripts/run_functional_evals.py \
8181
--verbose
8282
```
8383

84-
**What it checks** (12 eval prompts, 42 assertions total):
85-
86-
| Eval | Focus | Grader | Key assertions |
87-
| ------------------------------ | --------------------- | --------- | --------------------------------------------------------------------------------------------------------- |
88-
| 1. Transaction limits | MCP delegation | regex | Calls `awsknowledge`, cites 3,000 row limit, recommends batching |
89-
| 2. Multi-tenant schema | Correctness | regex | Uses `tenant_id`, `CREATE INDEX ASYNC`, no foreign keys, separate DDL txns |
90-
| 3. Index limits | MCP delegation | regex | Calls `awsknowledge`, cites 24 index limit, suggests alternatives |
91-
| 4. Python connection | Language routing | regex | Recommends DSQL Python Connector, IAM auth, 15-min token expiry, SSL |
92-
| 5. Column type change | DDL migration routing | regex | Table Recreation Pattern, DROP TABLE warning, batching, user confirmation |
93-
| 6. JSON column storage | Type guidance | LLM judge | Recommends `JSONB` (or `JSON`) as the column type for queryable structured data |
94-
| 7. Array storage | Type guidance | LLM judge | Flags `TEXT[]` / array column as unsupported, recommends storing the array as `JSONB` |
95-
| 8. INACTIVE cluster error | Troubleshooting | LLM judge | Identifies INACTIVE state, uses `aws dsql get-cluster` to poll until `ACTIVE`, retries afterwards |
96-
| 9. Backup on IDLE/INACTIVE | Troubleshooting | LLM judge | Identifies `FailedPrecondition`, connects to wake cluster to ACTIVE, retries backup |
97-
| 10. Loader stuck at 3K rec/s | Data loading | LLM judge | Identifies partition-constrained fresh table, advises to keep running, does NOT recommend more workers |
98-
| 11. Loader crash lost manifest | Data loading | LLM judge | Identifies /tmp as tmpfs, recommends --on-conflict do-nothing for recovery, --manifest-dir for prevention |
99-
| 12. Header row parse error | Data loading | LLM judge | Identifies missing --header flag, explains default behavior, recommends fix |
84+
**What it checks** (15 eval prompts, 54 assertions total):
85+
86+
| Eval | Focus | Grader | Key assertions |
87+
| ------------------------------ | --------------------- | --------- | --------------------------------------------------------------------------------------------------------------------- |
88+
| 1. Transaction limits | MCP delegation | regex | Calls `awsknowledge`, cites 3,000 row limit, recommends batching |
89+
| 2. Multi-tenant schema | Correctness | regex | Uses `tenant_id`, `CREATE INDEX ASYNC`, no foreign keys, separate DDL txns |
90+
| 3. Index limits | MCP delegation | regex | Calls `awsknowledge`, cites 24 index limit, suggests alternatives |
91+
| 4. Python connection | Language routing | regex | Recommends DSQL Python Connector, IAM auth, 15-min token expiry, SSL |
92+
| 5. Column type change | DDL migration routing | regex | Table Recreation Pattern, DROP TABLE warning, batching, user confirmation |
93+
| 6. JSON column storage | Type guidance | LLM judge | Recommends `JSONB` (or `JSON`) as the column type for queryable structured data |
94+
| 7. Array storage | Type guidance | LLM judge | Flags `TEXT[]` / array column as unsupported, recommends storing the array as `JSONB` |
95+
| 8. INACTIVE cluster error | Troubleshooting | LLM judge | Identifies INACTIVE state, uses `aws dsql get-cluster` to poll until `ACTIVE`, retries afterwards |
96+
| 9. Backup on IDLE/INACTIVE | Troubleshooting | LLM judge | Identifies `FailedPrecondition`, connects to wake cluster to ACTIVE, retries backup |
97+
| 10. Loader stuck at 3K rec/s | Data loading | LLM judge | Identifies partition-constrained fresh table, advises to keep running, does NOT recommend more workers |
98+
| 11. Loader crash lost manifest | Data loading | LLM judge | Identifies /tmp as tmpfs, recommends --on-conflict do-nothing for recovery, --manifest-dir for prevention |
99+
| 12. Header row parse error | Data loading | LLM judge | Identifies missing --header flag, explains default behavior, recommends fix |
100+
| 13. EF Core data layer setup | ORM routing (.NET) | LLM judge | Recommends Amazon.AuroraDsql.EntityFrameworkCore, Guid keys w/ gen_random_uuid(), app-layer FK, DsqlExecutionStrategy |
101+
| 14. .NET / C# support | Language routing | LLM judge | Confirms .NET support, recommends Npgsql connector for IAM auth and EF Core adapter |
102+
| 15. System diagnostics (W12) | AAS interpretation | LLM judge | Identifies the shifted wait event vs baseline, rules out load growth, no absolute-AAS claim, defers to Workflow 9 |
100103

101104
### Grader modes
102105

tools/evals/databases-on-aws/dsql/evals.json

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -170,6 +170,20 @@
170170
"Recommends the Amazon.AuroraDsql.EntityFrameworkCore adapter for EF Core users"
171171
],
172172
"llm_judge": true
173+
},
174+
{
175+
"id": 15,
176+
"prompt": "My Aurora DSQL cluster is performing differently than last week and I can't point to one bad query. I pulled the Active Average Sessions (db.active_sessions.avg) broken down by wait event for the same hour, this week vs last week. Last week: OnCpu 226, SequentialScanRead 89, UniqueConstraintCheck 89, Commit 74, everything else flat. This week: OnCpu 213, SequentialScanRead 336, UniqueConstraintCheck 87, Commit 73, everything else flat. What does this tell you and what should I do next?",
177+
"expected_output": "Interprets the data as a wait-event distribution shift: SequentialScanRead grew ~3.7x (from ~12% to ~35% of total AAS) while every other wait event, including OnCpu and Commit, stayed flat. Concludes this is NOT overall load growth (which would raise all events together) but a scan/plan regression. Does NOT treat any absolute AAS number as inherently bad — reasons from the change vs the temporal baseline. Recommends identifying the top SQL on SequentialScanRead and handing those queries off to per-query EXPLAIN analysis (Workflow 9). Does not recommend schema or index changes from CloudWatch data alone.",
178+
"files": [],
179+
"expectations": [
180+
"Identifies SequentialScanRead as the shifted wait event and quantifies the change relative to the baseline (roughly 3.7x, or a jump from ~12% to ~35% of total AAS)",
181+
"Concludes this is a scan/plan regression rather than proportional load growth, citing that OnCpu / Commit / other events stayed flat",
182+
"Does NOT claim any absolute AAS value is inherently problematic — reasons from change over time against the temporal baseline",
183+
"Recommends identifying the top SQL statements attributed to SequentialScanRead as the next step",
184+
"Defers per-query root cause to EXPLAIN / query plan analysis (Workflow 9) and does NOT recommend specific index or schema changes from the CloudWatch metrics alone"
185+
],
186+
"llm_judge": true
173187
}
174188
]
175189
}

tools/evals/databases-on-aws/dsql/trigger_evals.json

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,18 @@
5151
"query": "I think there's a bug in DSQL's EXPLAIN ANALYZE output — my Storage Lookup is reporting trillions of actual rows on a table that only has 300k rows. Can you diagnose this?",
5252
"should_trigger": true
5353
},
54+
{
55+
"query": "my Aurora DSQL cluster is performing differently than it was last week — can you diagnose what changed?",
56+
"should_trigger": true
57+
},
58+
{
59+
"query": "my dsql cluster feels slow but no single query stands out. how do I find where the time is going — wait events, active sessions, that kind of thing?",
60+
"should_trigger": true
61+
},
62+
{
63+
"query": "where is compute time being spent on my Aurora DSQL cluster? I want to see the AAS breakdown by wait event.",
64+
"should_trigger": true
65+
},
5466
{
5567
"query": "I need to load a 500MB CSV file into my Aurora DSQL table. What's the fastest way?",
5668
"should_trigger": true

0 commit comments

Comments
 (0)