Log-group tagging, CKAN civic-AI caveats, and JSON-RPC error-code fix#2
Merged
Merged
Conversation
…aset counts
Regression report from prod: when GPT-4o was asked "How many 311
service requests does Boston have in total?", the original (un-
enhanced) MCP forced a search_datasets→get_dataset→iterate-query_data
chain that surfaced per-year counts. The enhanced search_and_query
short-circuits that — picks the rolling NEW SYSTEM resource (~9,790
rows, the recent ~30-day operational view), presents it as canonical,
and the model reports 9,790 as "the total". 22 archives' worth of
data quietly missed.
Root cause: making one tool excellent at the 80% case ("give me
data") creates a path-of-least-resistance regression for the 20%
case ("give me a total"). Path of least resistance wins.
Fixes:
1. Prominent PARTIAL DATASET ANSWER block, prepended whenever
search_and_query auto-picked one of N queryable resources (i.e.
neither resource_name nor resource_index were provided). Tells
the model the answer is for one resource only and names the
three escape hatches: include_resource_totals=true, resource_name=
for a specific archive, or execute_sql with UNION ALL. Suppressed
when the model explicitly picked a resource — they got what they
asked for.
2. New include_resource_totals=true flag. When set, runs COUNT(*)
in parallel (asyncio.gather) against every queryable resource of
the matched dataset. Output prepends a "Per-resource totals"
block with a GRAND TOTAL line + per-resource breakdown. One
follow-up call gives the model the full breakdown that
previously took 22 sequential calls.
3. The existing siblings-list block is suppressed when
include_resource_totals is set, since the totals block already
lists all resources with counts.
Live verification (boston prod CKAN, this commit):
- search_and_query("311") with no flags → PARTIAL block at top,
9,790 rolling-window count, plus the siblings list and the path
to a real total.
- search_and_query("311", include_resource_totals=true) →
"GRAND TOTAL across 17 resources: 3402330" with per-year breakdown
(2011: 58262 ... 2026: 108033). Single call.
Tests: +3 (49 -> 52). Coverage: PARTIAL warning fires on auto-pick,
no warning when resource_name is explicit, parallel COUNTs sum to
correct grand total.
Add `Project = mcp-server` tags so the mcp-observability project can discover these log groups via the Resource Groups Tagging API. - Lambda log group (/aws/lambda/...): add tags block. - API Gateway access log group (/aws/apigateway/...-access): this group did not exist, so create it and wire access_log_settings on the prod stage to emit access logs to it. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Wire provenance, sample-size, freshness, type-fidelity, and field-validation guidance into the CKAN plugin's response formatters so the model gets explicit warnings instead of silently over-trusting open-data responses: - Provenance headers + echoed query params on every response (_wrap_response, _format_provenance_header, _params_repr). - Sample-size caveats: SMALL SAMPLE / single-record banners so counts and percentages are not generalized from tiny result sets. - Data freshness: flag datasets last edited beyond their update cadence (_parse_ckan_iso, _frequency_days, _format_freshness_caveat). - Stringly-typed columns: warn when date/number values sit in TEXT columns. - NULL-like frequency: DATA QUALITY caveat for columns that are mostly empty / "N/A" / "Unknown". - Field-name validation with "did you mean" suggestions against the real schema; search-ambiguity detection across candidate datasets. - search_and_query composite formatter for one-call keyword-to-rows. Adds 44 tests covering the new formatters. Full suite: 441 passed. Pre-existing work from the working tree; committed as-is (formatting normalized with ruff).
An unrecognized `method` was raised as a bare ValueError and caught by the
generic handler, which mapped it to -32603 ("Internal error"). Per JSON-RPC
2.0 that is a client error, not a server fault: well-behaved clients could
not tell "you called something that does not exist" apart from "the server
broke."
Introduce MethodNotFoundError and map it to -32601 ("Method not found") in
handle_request; everything else still maps to -32603. Also aligns
core/mcp_server.py with server/http_handler.py, which already uses -32601.
Found while smoke-testing the prod MCP server after the log-group tagging
change; not a regression from that work.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merges the
boston/security-hardening-and-aws-docsbranch intomain. Four commits:5a7ede2include_resource_totalsfor whole-dataset counts8256db3Project = "mcp-server"(Lambda + new API Gateway access log group) so the mcp-observability project can discover them via the Resource Groups Tagging API. Already deployed & verified in staging and prod.9479ea7search_and_querycomposite). +44 tests.6fd1ccf-32601(Method not found) for unknown methods instead of-32603(Internal error).Validation
pytest tests/)ruff check: clean on all changed filesruff format: applied; all changed files formattedterraform validateclean;terraform fmtclean on touched filesboston-ckan-mcp-staging) and prod (boston-opencontext-mcp-prod) log groups confirmed discoverable viaresourcegroupstaggingapi get-resources.Notes
9479ea7) were pre-existing work in the working tree, committed as-is with formatting normalized — not authored in the session that opened this PR..claude/,terraform/aws/tfplan,probe_*.json,live_*.json) were intentionally not committed.🤖 Generated with Claude Code