docs: cover listing consumers and changing rate limits in rpc-api-key skill

Author: spalladino

spartan/.claude/skills/create-rpc-api-key/SKILL.md

@@ -1,20 +1,31 @@
 ---
 name: create-rpc-api-key
-description: Add a new public RPC API key consumer to an Aztec RPC environment (mainnet or testnet) — mint the GCP Secret Manager secret with the helper script, wire it into Kong via Terraform, apply, verify it is live, and open the PR. Use when asked to add/create an RPC API key, onboard a new RPC client/consumer, or "give <someone> a mainnet/testnet rpc key".
+description: Manage public RPC API key consumers for an Aztec RPC environment (mainnet or testnet) — add a new consumer end-to-end (mint the GCP Secret Manager secret, wire it into Kong via Terraform, apply, verify, open the PR), list existing consumers and their real owners, and check or change a consumer's per-minute rate limit. Use when asked to add/create an RPC API key, onboard a new RPC client/consumer, "give <someone> a mainnet/testnet rpc key", or to list/inspect/raise/lower an RPC consumer's rate limit.
 ---
 
-# Create a public RPC API key
+# Manage public RPC API keys
 
-Adds one API key for the public RPC gateway. A key is a value stored in GCP Secret Manager; Terraform turns it into a Kong consumer that can call every keyed route in the environment. Externally the consumer is just `clientN`; the real recipient is recorded only as a private annotation on the secret.
+This runbook covers the full lifecycle of a public RPC gateway consumer: **creating** a key (Steps 1–6), **listing** who has keys, and **checking / changing** a consumer's rate limit.
+
+A key is a value stored in GCP Secret Manager; Terraform turns it into a Kong consumer that can call every keyed route in the environment. Externally the consumer is just `clientN`; the real recipient is recorded only as a private `client_name` annotation on the secret.
 
 The two RPC environments live in `spartan/terraform/deploy-rpc/environments/`:
 
 | Environment | dir | secret prefix | namespace / release prefix |
 |---|---|---|---|
 | mainnet | `environments/mainnet` | `mainnet-rpc-consumer` | `mainnet-rpc` / `mainnet` |
-| testnet | `environments/testnet` | (check that env's `CONSUMERS` entries) | `testnet-rpc` / `testnet` |
+| testnet | `environments/testnet` | `testnet-rpc-consumer` | `testnet-rpc` / `testnet` |
+
+The examples below use **mainnet**. For testnet, swap the dir, prefix, namespace, and context. Always read the target env's existing consumer entries and copy their exact prefix rather than assuming.
+
+## Two consumer-config shapes (mainnet vs testnet differ)
 
-The examples below use **mainnet**. For testnet, swap the dir, prefix, namespace, and context. Always read the target env's existing `CONSUMERS` entries and copy their exact prefix rather than assuming.
+The `CONSUMERS` map passed to the environment module is written differently in each env — read the target env's `main.tf` before editing:
+
+- **mainnet** — an **explicit map**, one `clientN` block per consumer, each with its own `rate_limit_minute`. This is where you set an individual mainnet consumer's limit.
+- **testnet** — **generated** from a `consumer_secret_names` list via a `for` comprehension that hardcodes `rate_limit_minute = 0` for every entry. So every testnet key is unlimited by construction, and there is no per-consumer limit knob unless you break a consumer out of the loop into its own explicit entry.
+
+**Rate-limit tiers.** `rate_limit_minute = 0` means **unlimited** (no rate-limit plugin is generated; see the `consumers_with_rate_limit` filter in `spartan/terraform/modules/rpc-gateway/main.tf`). A positive value generates a `rate-limiting` KongPlugin with `policy = "local"` and `limit_by = "consumer"` — **`local` means the limit is enforced per Kong pod**, so the effective ceiling is `rate_limit_minute × (number of Kong gateway pods)`. When `ALLOW_ANONYMOUS = true` (testnet), keyless callers fall back to an anonymous consumer capped at `ANONYMOUS_RATE_LIMIT_MINUTE` (default 300) **per IP, per pod**; mainnet sets `ALLOW_ANONYMOUS = false` so a key is mandatory there.
 
 ## Before you start — access and tooling
 
@@ -63,6 +74,17 @@ Edit `spartan/terraform/deploy-rpc/environments/<env>/main.tf` and append a `cli
 
 `rate_limit_minute = 0` means unlimited (no rate-limit plugin is generated). Use a positive value only if a cap is intended.
 
+**testnet variant.** testnet does not use an explicit map — it builds `CONSUMERS` from a `consumer_secret_names` list. Add the new secret name to that list instead:
+
+```hcl
+  consumer_secret_names = [
+    "testnet-rpc-consumer-client1",
+    "testnet-rpc-consumer-client2", # new
+  ]
+```
+
+Every entry inherits `rate_limit_minute = 0` (unlimited) from the comprehension. To give one testnet consumer a finite cap, it cannot stay in the uniform loop — break it out into its own explicit `CONSUMERS` entry (or merge an override) with the desired `rate_limit_minute`.
+
 ## Step 3 — plan and verify (the guardrail)
 
 ```bash
@@ -138,3 +160,69 @@ It is a bearer credential — share it over a secure channel, not plain chat/ema
 - Header form: `x-aztec-api-key: <KEY>` against `https://canonical.mainnet.rpc.aztec-labs.com/`
 
 The consumer is defined at the environment level, so the key authenticates on **every** keyed route in that environment (e.g. mainnet `v4`, `canonical`, and `v5` hosts) — one key, all routes.
+
+## Listing consumers and their owners
+
+The `clientN` → real-recipient mapping lives **only** in the `client_name` annotation on each GCP secret — it is never in the committed Terraform or in the cluster. To find who owns which key, list the secrets with their annotation:
+
+```bash
+# one env
+gcloud secrets list --filter="name:mainnet-rpc-consumer" \
+  --format="table(name.basename(), createTime, annotations.client_name)"
+
+# every RPC consumer across all envs (mainnet, testnet, staging, testnet-prover, eth-sepolia, …)
+gcloud secrets list --filter="name~rpc-consumer" \
+  --format="value(name.basename(), annotations.client_name)" | sort
+```
+
+To find a specific person's key, grep the annotation column — the owner may not have a key at all (in which case, on testnet, they are just using the anonymous tier and "bumping" them means creating a key):
+
+```bash
+gcloud secrets list --filter="name~rpc-consumer" --format="value(name.basename(), annotations.client_name)" | grep -i <name>
+```
+
+Cross-check against what is actually live in the cluster (the committed `main.tf` can lag behind live — always reconcile against `origin/merge-train/spartan`, not a possibly-stale checkout):
+
+```bash
+CTX=gke_testnet-440309_us-west1-a_aztec-gke-public
+kubectl --context "$CTX" -n <ns>-rpc get kongconsumer
+```
+
+## Checking a consumer's rate limit
+
+The declared limit is the consumer's `rate_limit_minute` in the env `main.tf` (`0` = unlimited). To read the **effective live** limit instead of the declared one:
+
+```bash
+CTX=gke_testnet-440309_us-west1-a_aztec-gke-public
+NS=testnet-rpc
+
+# does the consumer carry a rate-limit plugin annotation? (absent ⇒ unlimited)
+kubectl --context "$CTX" -n "$NS" get kongconsumer <release-prefix>-<key> -o jsonpath='{.metadata.annotations}'; echo
+
+# the per-consumer rate-limit plugin config, if any
+kubectl --context "$CTX" -n "$NS" get kongplugin <release-prefix>-<key>-rpc-rate-limit -o jsonpath='{.config}'; echo
+
+# the anonymous (keyless) limit, per route — testnet only
+kubectl --context "$CTX" -n "$NS" get kongplugin <release-prefix>-<route>-anonymous-rpc-rate-limit -o jsonpath='{.config}'; echo
+
+# Kong pod count — the local policy multiplies the limit by this
+kubectl --context "$CTX" -n "$NS" get pods | grep kong-gateway
+```
+
+Remember `policy = "local"`: the real ceiling is `minute × (Kong gateway pod count)`. With a single gateway pod, the number in the plugin is the number the client sees.
+
+## Changing a consumer's rate limit
+
+Editing `rate_limit_minute` for an existing consumer, then `plan`/`apply` exactly as in Steps 3–4. **The Step 3 guardrail still applies**: the plan must touch only that one consumer's rate-limit plugin and its `KongConsumer` annotation — nothing else. Expected plan deltas by transition:
+
+| Change | Expected plan |
+|---|---|
+| `0` → `N` (add a cap) | `1 to add, 1 to change` — creates the rate-limit `KongPlugin`, annotates the consumer to use it |
+| `N` → `M` (both > 0) | `1 to change` — just the plugin's `minute` |
+| `N` → `0` (remove the cap) | `1 to change, 1 to destroy` — drops the plugin, removes the consumer annotation |
+
+Notes:
+
+- **testnet caveat**: a testnet consumer generated by the `consumer_secret_names` loop is uniformly `rate_limit_minute = 0`. You cannot raise *one* testnet consumer above/below the others without pulling it out of the loop into an explicit `CONSUMERS` entry (see the testnet variant under Step 2). If the intent is simply "more than the 300/min anonymous tier", giving them a key already makes them unlimited — no per-consumer plugin needed.
+- **Raising the anonymous tier** (testnet, keyless users) is a different lever: set `ANONYMOUS_RATE_LIMIT_MINUTE` on the testnet env module (default 300). This affects **every** keyless caller, not one person.
+- Commit message convention for a limit change is a normal `chore:`/`fix:` describing the change (not `chore: add new client`), on base `merge-train/spartan`. Do not name the private owner in the PR.