merge: PR #610 — kubelet-owned volume permissions + paid-smoke hardening

Author: bussyjd

.agents/skills/obol-stack-dev/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: obol-stack-dev
-description: Obol Stack development and QA runbook. Use when working on obol-stack flows, x402 seller/buyer tests, live Base Sepolia OBOL smoke, Anvil fork regressions, ERC-8004 registration, LiteLLM paid routing, release-smoke, cloudflared, Renovate image bumps, or remote QA worktrees.
+description: CLI-first Obol Stack development and QA runbook. Use when working on obol-stack lifecycle, obol CLI surfaces, x402 seller/buyer tests, live Base Sepolia OBOL smoke, Anvil fork regressions, ERC-8004 registration, LiteLLM paid routing, release-smoke, cloudflared, Renovate image bumps, or remote QA worktrees.
 metadata:
   version: "3.1.0"
   domain: infrastructure
@@ -12,6 +12,19 @@ metadata:
 
 Operational router. Load only the reference for the task. **Do not delegate understanding** — read the relevant reference yourself; subagents lose context the next reference would have given them.
 
+## CLI-First Rule
+
+Prefer the supported `obol` CLI surface for every stack operation: `obol stack`,
+`obol model`, `obol agent`, `obol sell`, `obol buy`, `obol network`, `obol
+tunnel`, and `obol kubectl` for inspection. Do not create ad-hoc shell scripts
+or verifier scripts. Do not bypass CLI behavior with direct script execution,
+raw `kubectl` mutation, or ConfigMap surgery when an `obol` command exists.
+
+Existing repository flow scripts are release-gate artifacts, not the default QA
+interface. Use them only when the user explicitly asks for release-smoke/full
+flow validation or when no CLI equivalent exists; otherwise decompose checks
+into CLI commands plus `obol kubectl` evidence.
+
 ## Reference Router
 
 | Need | Read |
@@ -28,11 +41,11 @@ Operational router. Load only the reference for the task. **Do not delegate unde
 ## First Actions on Any Task
 
 1. Read existing files before changing anything.
-2. Use repo flows/helpers; don't invent ad-hoc scripts.
+2. Use `obol` CLI and `obol kubectl` first; do not invent or run ad-hoc scripts.
 3. New worktree per remote QA run.
 4. Never write hostnames, personal paths, passwords, private keys, or raw tokens into skill files, PR text, or commit messages.
 5. Validate with the narrowest command set that covers the change.
-6. On a dev branch (anything not at the latest release tag), set `OBOL_DEVELOPMENT=true` for `obolup.sh` and `obol stack up`. Without it, `obolup.sh` downloads the released binary and your branch changes never run. Replace the `go run` wrapper with a real binary before running flows (`go build -o .workspace/bin/obol ./cmd/obol`) — backgrounded port-forwards in flows false-FAIL if the wrapper is recompiling.
+6. On a dev branch (anything not at the latest release tag), set `OBOL_DEVELOPMENT=true` for `obolup.sh` and `obol stack up`. Without it, `obolup.sh` downloads the released binary and your branch changes never run. Replace the `go run` wrapper with a real binary before long QA (`go build -o .workspace/bin/obol ./cmd/obol`) so every CLI call uses the same branch build.
 
 ## Critical Invariants
 
@@ -42,11 +55,11 @@ OBOL_TOKEN_BASE_SEPOLIA=0x0a09371a8b011d5110656ceBCc70603e53FD2c78
 # Source of truth: ObolNetwork/obol-stack#447
 ```
 
-**Buyer wallet (Bob)**: deterministic 2nd-derived key from `.env REMOTE_SIGNER_PRIVATE_KEY`. Flows 11/13/14 must pre-seed Bob's remote-signer before Bob's `stack up`, then assert `bobSigner == BOB_WALLET`. **Do not** transfer funds to a generated signer to make the test pass.
+**Buyer wallet (Bob)**: deterministic 2nd-derived key from `.env REMOTE_SIGNER_PRIVATE_KEY`. Flows 11/13/14 must seed Bob's remote-signer through `obol wallet import` after Bob's stack and LLM route are up, then assert `bobSigner == BOB_WALLET`. **Do not** transfer funds to a generated signer to make the test pass.
 
 **Token/auth**: use `obol agent auth --runtime <runtime> obol-agent`. **Never** `obol hermes token obol-agent` — it can print CLI usage text and poison the Bearer token.
 
-**Payment assertion**: don't bypass the agent buy step with a direct script exec. If the agent times out, diagnose Hermes/LiteLLM/model routing — don't relax the assertion. Required evidence: `PurchaseRequest Ready=True` + paid HTTP 200 + on-chain `Transfer` + exact balance deltas.
+**Payment assertion**: don't bypass the CLI/agent buy path with direct script exec. Prefer `obol buy inference`, `obol sell ...`, `obol agent ...`, and `obol kubectl` status checks. If the agent times out, diagnose Hermes/LiteLLM/model routing — don't relax the assertion. Required evidence: `PurchaseRequest Ready=True` + paid HTTP 200 + on-chain `Transfer` + exact balance deltas.
 
 **QA LLM**: full seller/buyer QA must route Alice and Bob through `OBOL_LLM_ENDPOINT` (OpenAI-compatible vLLM or llama.cpp on the QA host). Default `OBOL_LLM_MODEL=qwen36-deep` (27B-class). The smaller `qwen36-fast` (~4B) was the previous default but flakes on the long single-shot agent-buy prompt at flow-13/14 step 46 — see the retry-wrapper rationale in `flows/lib-dual-stack.sh::agent_buy_with_retry`. Sequence: `obol model setup custom` → `obol model prefer` → one `obol model sync`. Local Ollama and cloud-fallback are **not** acceptable green substitutes for full-flow QA.
 
@@ -70,7 +83,7 @@ When the smoke gate goes red, check these first — each was a multi-hour debug:
 | First request after fresh verifier deploy returns empty body | Traefik HTTPRoute is wired but verifier's serviceoffer-source watcher hasn't loaded the route yet. | `flows/flow-07-sell-verify.sh` + `flows/flow-08-buy.sh` — wrap 402-body fetch in 12×5s retry loop. |
 | facilitator arm64 image runs amd64 binary | Was an `ObolNetwork/x402-rs` prom-overlay arm64 manifest packaging bug. | **Fixed upstream**: `ObolNetwork/x402-rs#3` (merged 2026-05-13, `668b7bb`) dropped the redundant `--platform=$BUILDPLATFORM` pin from the prom-overlay builder stage. Registry image republished; arm64 manifest now ships an aarch64 ELF (digest `sha256:b209345c…`). The `X402_FACILITATOR_SKIP_PULL` knob has been removed from `flows/lib.sh`. |
 
-**Diagnosis pattern**: a 503 from the verifier or 404 from a paid route almost never means the verifier is bad — it usually means the deployed image isn't what you think it is, the chain id form mismatched, or the upstream wasn't reachable. Confirm the running image first (`kubectl get deploy -n x402 x402-verifier -o jsonpath='{.spec.template.spec.containers[*].image}'`) before diving into x402 logic.
+**Diagnosis pattern**: a 503 from the verifier or 404 from a paid route almost never means the verifier is bad — it usually means the deployed image isn't what you think it is, the chain id form mismatched, or the upstream wasn't reachable. Confirm the running image first (`obol kubectl get deploy -n x402 x402-verifier -o jsonpath='{.spec.template.spec.containers[*].image}'`) before diving into x402 logic.
 
 ## Force a Fresh Local Image Build
 
@@ -123,7 +136,7 @@ Examples:
 ## Pre-Push Local Checks
 
 ```bash
-bash -n flows/*.sh                                                    # shell syntax
+bash -n flows/*.sh                                                    # only when flow scripts changed
 git diff --check                                                      # whitespace/conflict markers
 jq empty renovate.json                                                # JSON valid
 helm lint internal/embed/infrastructure/cloudflared
@@ -135,6 +148,6 @@ go test ./cmd/obol ./internal/x402/... ./internal/defaults/... -count=1   # touc
 
 ## Editing This Skill
 
-Do: keep `SKILL.md` short and operational; one fact lives in one place; references one hop from `SKILL.md`. Inline shell snippets directly in the markdown — don't ship parallel implementations of logic that already lives in `flows/lib.sh` or `internal/...`.
+Do: keep `SKILL.md` short and operational; one fact lives in one place; references one hop from `SKILL.md`. Prefer CLI examples. Do not add bundled scripts or parallel implementations of logic that belongs in `obol` CLI, `flows/lib.sh`, or `internal/...`.
 
 Don't: README-style prose; duplicate the same procedure in `SKILL.md` and references; bury safety constraints below examples; copy host-specific names, credentials, or logs.

.agents/skills/obol-stack-dev/references/dev.md

@@ -1,5 +1,13 @@
 # Dev Environment & CLI
 
+## Operating Mode
+
+Use the product surface first: `obol ...` commands for lifecycle and mutations,
+`obol kubectl ...` for Kubernetes evidence, and `curl` only for endpoint probes.
+Do not write custom `.sh` helpers for stack checks. Existing `flows/*.sh` are
+release-gate artifacts; reach for them only when the user asks for full
+release-smoke or a named flow regression.
+
 ## Bootstrap
 
 Dev mode uses `.workspace/` instead of XDG dirs. Without `OBOL_DEVELOPMENT=true`, `obolup.sh` downloads the released binary and your branch changes never run.
@@ -24,7 +32,7 @@ go build -o .workspace/bin/obol ./cmd/obol
 .workspace/bin/obol version
 ```
 
-**Always replace the wrapper before running flows.** `obolup.sh` with `OBOL_DEVELOPMENT=true` installs a `go run -a` wrapper at `.workspace/bin/obol`. It recompiles on every invocation; backgrounded port-forwards (e.g. `flow-06` step 15) false-FAIL because the listener isn't ready in 5–8 seconds.
+**Always replace the wrapper before long QA.** `obolup.sh` with `OBOL_DEVELOPMENT=true` installs a `go run -a` wrapper at `.workspace/bin/obol`. It recompiles on every invocation and can make repeated CLI calls or port-forwards look flaky.
 
 ```bash
 mv .workspace/bin/obol .workspace/bin/obol.wrapper
@@ -75,6 +83,9 @@ go build -o .workspace/bin/obol ./cmd/obol      # rebuild after every code chang
 | passthrough | `kubectl` `helm` `helmfile` `k9s` |
 | meta | `update` `upgrade` `version` |
 
+Use passthrough tools through `obol` (`obol kubectl ...`, `obol helm ...`) so
+the active stack kubeconfig and dev paths are selected consistently.
+
 ## `obol sell http` — easy-to-misuse flag set
 
 Common mistakes: `--model`, `--pay-to`, and `--network` do **not** exist on `sell http`.

.agents/skills/obol-stack-dev/references/integration-testing.md

@@ -61,7 +61,7 @@ go test -tags integration -v -run TestBDDIntegration -timeout 10m ./internal/x40
 |---|---|---|
 | `obol openclaw sync` (first deploy) | 5–15 s | 60 s |
 | `obol openclaw sync` (re-deploy, no changes) | 2–5 s | 60 s |
-| Pod startup | 10–60 s | 180 s (`kubectl wait`) |
+| Pod startup | 10–60 s | 180 s (`obol kubectl wait`) |
 | Port-forward ready | 1–10 s | 30 s |
 | Chat completion (Ollama) | 1–30 s | 90 s |
 | Chat completion (cloud) | 2–10 s | 90 s |

.agents/skills/obol-stack-dev/references/llm-routing.md

@@ -18,14 +18,14 @@ LiteLLM lives in the `llm` namespace, port 4000. Agents and skills in other name
 
 ## Reaching Services from the Mac Host
 
-Only routes published through Traefik are reachable at `http://obol.stack:8080/`. Everything else needs `kubectl port-forward`:
+Only routes published through Traefik are reachable at `http://obol.stack:8080/`. Everything else needs `obol kubectl port-forward`:
 
 | Service | Access |
 |---|---|
 | Traefik ingress (frontend, eRPC, x402 routes) | `http://obol.stack:8080/...` |
-| LiteLLM | `kubectl port-forward svc/litellm 14000:4000 -n llm` then `http://127.0.0.1:14000` |
-| x402-buyer sidecar (no Service — pod only) | `kubectl port-forward -n llm <litellm-pod> 18402:8402` then `http://127.0.0.1:18402` |
-| OpenClaw instance | `kubectl port-forward -n openclaw-<id> svc/openclaw 18789:18789` |
+| LiteLLM | `obol kubectl port-forward svc/litellm 14000:4000 -n llm` then `http://127.0.0.1:14000` |
+| x402-buyer sidecar (no Service — pod only) | `obol kubectl port-forward -n llm <litellm-pod> 18402:8402` then `http://127.0.0.1:18402` |
+| OpenClaw instance | `obol kubectl port-forward -n openclaw-<id> svc/openclaw 18789:18789` |
 
 `http://obol.stack:8080/v1/...` does **not** hit LiteLLM — Traefik has no `/v1` route and returns the frontend 404. The `x402-buyer` sidecar is **distroless** — no `wget`/`curl`/shell. Always port-forward, never `kubectl exec`.
 
@@ -63,7 +63,9 @@ obol model list      # confirm the custom entry is the only local model
 obol model status    # provider state
 ```
 
-The flow scripts (`flows/lib.sh::route_llm_via_obol_cli`) wrap this exact sequence behind `OBOL_LLM_ENDPOINT` / `OBOL_LLM_MODEL` / `OBOL_LLM_API_KEY` env vars so smoke tests target a GPU host without burning host CPU on local Ollama.
+Release flow internals wrap this sequence behind `OBOL_LLM_ENDPOINT` /
+`OBOL_LLM_MODEL` / `OBOL_LLM_API_KEY`. For manual QA, run the `obol model ...`
+commands directly.
 
 ## Paid Routing (`paid/<remote-model>`)
 
@@ -97,7 +99,7 @@ maintain          # alias for `process --all`
 
 ### Endpoint URLs inside pods vs the Mac host
 
-`obol.stack:8080` only resolves on the Mac host (via the DNS resolver). From inside any pod (buy.py, kubectl exec, anything), use the Traefik cluster-internal address:
+`obol.stack:8080` only resolves on the Mac host (via the DNS resolver). From inside any pod (release flow internals, `obol kubectl exec`, anything), use the Traefik cluster-internal address:
 
 - Host:   `http://obol.stack:8080/services/<name>/...`
 - In-pod: `http://traefik.traefik.svc.cluster.local/services/<name>/...`
@@ -108,4 +110,4 @@ maintain          # alias for `process --all`
 
 ## When LiteLLM Restart is Needed (Fallback Only)
 
-The validated happy path is `buy.py buy` / `process --all` / same-name top-up **without** a manual LiteLLM restart. The hot-add/hot-delete plus buyer reload normally makes `paid/<model>` appear/disappear in place. Restart only as a fallback investigation step if the route doesn't appear after the controller reconciled and the buyer reports the upstream.
+The validated user path is `obol buy inference` / same-name top-up **without** a manual LiteLLM restart. The embedded `buy.py` path is for release flow internals or skill debugging. The hot-add/hot-delete plus buyer reload normally makes `paid/<model>` appear/disappear in place. Restart only as a fallback investigation step if the route doesn't appear after the controller reconciled and the buyer reports the upstream.

.agents/skills/obol-stack-dev/references/paid-flows.md

@@ -1,6 +1,10 @@
 # Paid Flows (Live OBOL + Anvil Fork)
 
-Use this for `flow-11`, `flow-13`, `flow-14`, release-smoke OBOL gating, and demo validation.
+Use this for release-smoke OBOL gating, named flow regressions, and demo
+validation. For ordinary QA, prefer CLI-first checks with `obol sell`, `obol
+buy`, `obol model`, and `obol kubectl`. Use `flows/*.sh` only when the user asks
+for release-smoke/full-flow validation or the regression is specifically inside
+a named flow.
 
 ## Flow Selection
 
@@ -23,7 +27,7 @@ Keep these explicit in the run command. Don't hide live/fork behind one selector
 
 ## Wallet Invariant
 
-Both Alice (seller/register) and Bob (buyer) derive from a single `.env REMOTE_SIGNER_PRIVATE_KEY`. Bob is the deterministic 2nd-derived key. The flow pre-seeds Bob's remote-signer with this key **before** Bob's `stack up` and asserts `bobSigner == BOB_WALLET`.
+Both Alice (seller/register) and Bob (buyer) derive from a single `.env REMOTE_SIGNER_PRIVATE_KEY`. Bob is the deterministic 2nd-derived key. The flow seeds Bob's remote-signer with this key through `obol wallet import` after Bob's stack and LLM route are up, then asserts `bobSigner == BOB_WALLET`.
 
 **Do not** transfer funds to a generated signer to make the test pass. Keep live OBOL funding on the deterministic Bob address. Don't infer the canonical pair from balances on older duplicate token deployments.
 
@@ -85,15 +89,18 @@ Do **not** treat raw `X-PAYMENT` through Traefik ForwardAuth as a supported prod
 Fixed automatically: `obol stack up` calls `x402verifier.PopulateCABundle` after infra deploy; `obol sell http` calls it before creating the ServiceOffer. Manual repopulate:
 
 ```bash
-kubectl create configmap ca-certificates -n x402 \
+obol kubectl create configmap ca-certificates -n x402 \
   --from-file=ca-certificates.crt=/etc/ssl/cert.pem \
-  --dry-run=client -o yaml | kubectl replace -f -
+  --dry-run=client -o yaml | obol kubectl replace -f -
 ```
 
 ## Quick Full-Cycle Smoke
 
-1. **Unpaid gate**: POST seller route without `X-PAYMENT` → expect 402 + accepts requirements.
-2. **Buy auths**: `buy.py buy <name> --endpoint <url> --model <id> --count N` → expect PurchaseRequest `Ready` and sidecar `/status` shows `remaining > 0`.
-3. **Paid call**: LiteLLM request with model `paid/<remote-model>` → expect 200.
-4. **Spend proof**: sidecar `/status` shows `remaining −1`, `spent +1`.
-5. **Auto-refill**: create with `--auto-refill ...`, run `buy.py process --all`, confirm the loop only signs when live `/status` is at or below threshold.
+1. **Configure model**: `obol model setup custom --endpoint <url>/v1 --model <id>`; then `obol model prefer <id>` and `obol model sync`.
+2. **Sell**: use `obol sell inference` or `obol sell demo <type>`; wait for `obol sell status <name> -n <ns>` to show `Ready=True`.
+3. **Unpaid gate**: `obol sell test <name> -n <ns>`; expect HTTP 402 + accepts requirements.
+4. **Buy**: use `obol buy inference <seller-url> --yes --count <N>` when testing buyer flow through the CLI. Confirm `PurchaseRequest Ready=True` with `obol kubectl get purchaserequest -A`.
+5. **Paid call and spend proof**: call LiteLLM with `paid/<model>` and verify HTTP 200, sidecar `/status` spend counters, and on-chain balance deltas when the test is live-settlement gated.
+
+Direct `buy.py` execution is reserved for existing release flow internals or
+debugging a bug in that embedded skill itself.