Skip to content

Add cloud service query with auto-provisioned read-only key#142

Open
sdairs wants to merge 6 commits into
mainfrom
feat-cloud-service-query-autoprovision
Open

Add cloud service query with auto-provisioned read-only key#142
sdairs wants to merge 6 commits into
mainfrom
feat-cloud-service-query-autoprovision

Conversation

@sdairs
Copy link
Copy Markdown
Collaborator

@sdairs sdairs commented May 11, 2026

Summary

  • Adds cloud service query — runs SQL over HTTP via the ClickHouse Cloud Query API, no local clickhouse binary or service password required.
  • On cloud service create, auto-provisions a dedicated read-only API key, binds it to the service's query endpoint with sql_console_read_only, and persists key_id/key_secret/endpoint_id in .clickhouse/credentials.json. Opt out via --no-enable-query; lazy provisioning is on by default for cloud service query (disable with --no-auto-enable). cloud service delete clears the stored key.
  • Promotes run_query into clickhouse-cloud-api so the library's live integration test exercises the full Query API flow (create key → upsert endpoint with read-only role → SELECT 1 over HTTP → delete key). Adds api-key tracking to CleanupRegistry so leaks can't outlive a failed run.
  • Marks ApiKeyPostRequest.roles and ApiKeyPostRequest.hashData as Option<T> with OPTIONALITY_EXEMPTIONS entries — the spec's description-heuristic classifies them as required, but the live API treats them as opt-in (roles is deprecated with minLength=1; hashData is opt-in per its own response-side spec note). Resolves three CI failures uncovered by the new integration phase.
  • Uses the API key resource UUID (key.id) for endpoint binding and management calls; the credential id (keyId) is reserved for query-time HTTP auth. This was the third CI failure: management endpoints reject the credential id with Invalid API key id.

Test plan

  • cargo build workspace
  • cargo test (all unit + spec-coverage tests pass; field_optionality_matches_spec accepts the two new exemptions and reports no stale entries)
  • Live integration test (cloud_service_crud_lifecycle) — Query API phase now passes end-to-end against ClickHouse Cloud
  • Manual: clickhousectl cloud service create → clickhousectl cloud service query --query 'SELECT 1' against a fresh service
  • Manual: clickhousectl cloud service create --no-enable-query → clickhousectl cloud service query triggers lazy provisioning; --no-auto-enable short-circuits with a helpful error

🤖 Generated with Claude Code

sdairs and others added 4 commits May 11, 2026 15:59
@sdairs @claude
Add cloud service query with auto-provisioned read-only key
b63e7d9 
Runs SQL over HTTP via the ClickHouse Cloud Query API — no local
clickhouse binary and no service password required. On `cloud service
create` we now also create a dedicated read-only API key, bind it to the
service's query endpoint with role `sql_console_read_only`, and persist
key_id/key_secret/endpoint_id in `.clickhouse/credentials.json` under a
new `service_query_keys` map. `cloud service query` looks up the stored
key and posts to queries.clickhouse.cloud (overridable via
`CLICKHOUSECTL_QUERY_HOST`). Missing-key calls auto-provision lazily;
`--no-auto-enable` opts out, and `cloud service create --no-enable-query`
skips the create-time hook. SQL precedence is `--query` >
`--queries-file` > stdin; default format is PrettyCompact on a TTY,
TabSeparated when piped. `cloud service delete` clears the stored key.

The credentials struct now treats `api_key`/`api_secret` as optional so
OAuth-only users can still hold per-service query keys.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@sdairs @claude
Make Query API the canonical query path in the integration test
0e16353 
Moves the data-plane `run_query` call from clickhousectl into the
shared `clickhouse-cloud-api` crate (as `Client::run_query`) so the
library integration test can exercise it directly without depending on
the CLI binary. The CLI now delegates to it; `RunQueryRequest` and
`CloudClient::run_service_query` are removed.

Adds a new "Query API Endpoint" phase to the live cloud lifecycle test,
sitting between Provision and Stop/Start. The phase creates a
read-only API key, upserts the query endpoint with
`sql_console_read_only`, runs `SELECT 1` over HTTP, asserts the
response, and deletes the key. This is the same flow `cloud service
query` uses end-to-end — `cloud service client` is on a deprecation
path. Polling tolerates a brief endpoint propagation lag.

`CleanupRegistry` gains api-key tracking so leaked keys can't outlive a
failed run. The spec-coverage test gets a small allowlist for client
methods (currently just `run_query`) that intentionally don't map to an
OpenAPI operation.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@sdairs @claude
Set legacy roles=["query_endpoints"] on auto-provisioned API keys
c7776d2 
The cloud API enforces `minLength=1` on `ApiKeyPostRequest.roles` even
though the field is marked deprecated in the OpenAPI spec. With
`roles=[]` and an empty `assignedRoleIds`, key creation fails with
`request body.roles array length < minLength`.

Set `roles=["query_endpoints"]` — the legacy role intended for keys
scoped to Query API endpoint access — in both `service_query::
ensure_service_query_setup` and the integration test's key-create
step. The functional read-only constraint still comes from the
`sql_console_read_only` role on the query endpoint binding.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@sdairs @claude
Fix Query API auto-provisioning to match Cloud API contract
c1551f2 
Three CI integration-test failures shared a single theme — the
OpenAPI spec's description-heuristic marks fields as "required" that
the live API treats as optional, and the credential `keyId` is not
the same value as the API key resource UUID. All three are fixed at
the model level so callers don't have to work around them.

1. `ApiKeyPostRequest.roles` is deprecated but still enforces
   `minLength=1`, so the bare `Vec<String>` default serialized as
   `"roles":[]` and was rejected. Changed to `Option<Vec<String>>`
   with an `OPTIONALITY_EXEMPTIONS` entry; callers using
   `assignedRoleIds` now pass `None` and the field is omitted.

2. `ApiKeyPostRequest.hashData` is opt-in (the API generates the key
   when omitted, per the spec's response-side description) but the
   model required it, yielding `Not a sha256sum: keyIdHash` for the
   default value. Changed to `Option<ApiKeyHashData>` with an
   exemption; the CLI's `parse_api_key_hash_data` already returned
   `Option`, so the `.unwrap_or_default()` workaround is dropped.

3. `openApiKeys` in the endpoint binding and the management endpoints
   (GET/DELETE `/keys/{id}`) accept the API key's resource UUID
   (`key.id`), not the short credential id (`keyId`) used for query
   auth. `ensure_service_query_setup` and the integration test now
   send the UUID for binding and cleanup, and keep the credential
   pair only for `run_query`.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@sdairs sdairs requested a review from iskakaushik as a code owner May 11, 2026 17:57
@sdairs sdairs temporarily deployed to cloud-integration May 11, 2026 17:57 — with GitHub Actions Inactive
@sdairs @claude
Cover pre-enable failure and re-upsert idempotency in Query API phase
2a6905c 
Two regressions in the Query API contract would slip past the existing
integration coverage:

1. The control plane silently honouring queries against a service with
   no endpoint binding. Adds `query before endpoint enabled fails`
   between key creation and the first upsert: `run_query` must return
   a 4xx (not pinned to a specific status — 401/403/404 all evidence
   "not enabled"). A 2xx fails the test.

2. A re-upsert with the same body rotating the endpoint resource id
   or stripping the binding. Captures the first upsert result, then
   adds `re-upsert query endpoint is idempotent`, which posts the same
   body again and asserts `endpoint.id`, the `openApiKeys` entry and
   the `sql_console_read_only` role all survive. A follow-up
   `SELECT 1 still works after re-upsert` confirms the original
   credentials remain valid afterwards.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@sdairs sdairs temporarily deployed to cloud-integration May 11, 2026 19:50 — with GitHub Actions Inactive
@sdairs
Copy link
Copy Markdown
Collaborator Author

sdairs commented May 11, 2026

closes

#124
#123
#122

@sdairs @claude
Grant auto-provisioned Query API key read+write on its service
48139ff 
Switch the endpoint binding role from sql_console_read_only to
sql_console_admin so users can INSERT and run DDL through
`cloud service query`. Scoping stays per-service: the binding (not the
API key's org roles) is what grants access, so the key still cannot
reach any other service in the org.

Integration test now CREATEs a table, INSERTs rows, and verifies the
SELECT sum to catch a regression if the binding ever silently
downgrades to read-only.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@sdairs sdairs temporarily deployed to cloud-integration May 11, 2026 20:44 — with GitHub Actions Inactive
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@iskakaushik iskakaushik iskakaushik approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@sdairs @iskakaushik