fix(pvc): legacy hostPath-PV compat, scoped Reloader, flow alignment with #614

Applies the verified findings from the cross-review against PR #614 (every
item adversarially confirmed against the refs; union merge-tree clean).

PVC / upgrade path:
- llm.yaml: restore container-level runAsUser/runAsGroup 1000 on x402-buyer.
  Clusters upgraded in place from <= rc12 keep hostPath-typed PVs where
  kubelet skips fsGroup; their /state dir is 1000:1000 with consumed.json
  written 0600 by UID 1000 — a 65532 sidecar cannot read it, Fatalf's on
  `load state`, and takes every paid/<model> route down. On fresh local-type
  PVs the explicit UID is harmless (fsGroup 65532 grants group access).
  embed_buyer_state_test.go updated to pin the new contract.
- plans/volume-permission-hardening.md: new "Upgrading from <= v0.10.0-rc12"
  section — supported path is cluster recreation (wallet backup/restore),
  with a documented k3d chown escape hatch. troubleshooting.md gets the
  symptom->fix entry. The Hermes half of the legacy-PV breakage cannot be
  patched at runtime without reintroducing the chown machinery this PR
  removes, so it is a documented breaking change instead.

Paid-route availability:
- llm.yaml: Reloader annotation narrowed to litellm-config only. The buyer
  ConfigMaps (x402-buyer-config/x402-buyer-auths) are rewritten by the
  controller on every buy, top-up, auto-refill, and tombstone cleanup;
  with strategy Recreate + 1 replica the previous annotation bounced the
  entire inference gateway (all Hermes traffic, in-flight SSE streams) on
  every purchase event, inverting CLAUDE.md pitfall 7 (restart is fallback,
  not the default buy path). The buyer hot-reloads via /admin/reload.
  stack_test.go updated to pin litellm-config-only.

Flow alignment with #614:
- lib.sh: `stack down` -> `stack down --yes` in reset_flow_workspace. #614's
  flow-16 (now last in the single-stack array) intentionally leaves a live
  agent offer; without --yes the non-TTY ConfirmRunningServicesLoss gate
  refuses, graceful down is silently skipped (`|| true`), and teardown
  degrades to the raw k3d-delete fallback on every release-smoke run.
- flow-11: post-register Ready poll 120s -> 300s to match flow-14's identical
  live-Base-Sepolia chain-watch path (pitfall 13 free-tier RPC throttling).

Known follow-ups (not in this commit): flow-08 buy-retry top-up vs exactly-N
assertions on rare partial failures; flow-11 lacks flow-14's remote-signer
rolled guard; aztec PVC has no permission story (runs as root today);
post-merge controller repin so released sub-agents pick up this PR's
UID-1000 render (tracked in #614's pin-test note).

Author: bussyjd

.agents/skills/obol-stack-dev/references/troubleshooting.md

@@ -32,6 +32,17 @@ data alone. Use `FLOW_FORCE_PURGE_DATA=true` or `RELEASE_SMOKE_FORCE_PURGE_DATA=
 only when the operator explicitly wants full persistent-data deletion and has an
 interactive sudo path.
 
+### Hermes/x402-buyer EACCES or crashloop after upgrading a pre-v0.10.0 cluster
+
+PVs provisioned before v0.10.0 are hostPath-typed — kubelet skips fsGroup
+ownership there, and the v0.10.0 pods (UID 1000, no root chown init) cannot
+read legacy data owned 10000:10000. Symptoms: Hermes gateway crashloops on
+state.db / config.yaml; x402-buyer exits `load state:` at startup, killing
+every `paid/<model>` route. Fix: recreate the cluster (`stack down` →
+`purge -f` → `init` → `up`; back up agent wallets first), or for k3d chown
+the PV backing dirs to 1000:1000 from inside the node and restart the pods.
+See plans/volume-permission-hardening.md "Upgrading from <= v0.10.0-rc12".
+
 ### k3d port 80 privileged on macOS
 
 Always use `http://obol.stack:8080/`, not `http://obol.stack/`. Port 8080 maps to the same Traefik LB.

flows/flow-11-dual-stack.sh

@@ -1054,7 +1054,10 @@ if [ "$register_rc" -ne 0 ]; then
 fi
 pass "obol sell register issued"
 
-poll_step_grep "Alice: ServiceOffer Ready" "True" 24 5 \
+# 300s to match flow-14's post-register poll: Ready requires the controller's
+# Base Sepolia chain watch (via eRPC) to observe the register tx, and the
+# free-tier RPC throttling in pitfall 13 makes 120s intermittently tight.
+poll_step_grep "Alice: ServiceOffer Ready" "True" 60 5 \
     alice kubectl get serviceoffers.obol.org alice-inference -n llm \
         -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}'
 

flows/lib.sh

@@ -488,7 +488,7 @@ reset_flow_workspace() {
         OBOL_CONFIG_DIR="$dir/config" \
         OBOL_BIN_DIR="$dir/bin" \
         OBOL_DATA_DIR="$dir/data" \
-            run_with_timeout 120 "$obol_cmd" stack down >/dev/null 2>&1 || true
+            run_with_timeout 120 "$obol_cmd" stack down --yes >/dev/null 2>&1 || true
         if [ "$force_purge" = "true" ]; then
             OBOL_DEVELOPMENT=true \
             OBOL_NONINTERACTIVE=true \

internal/embed/embed_buyer_state_test.go

@@ -92,9 +92,12 @@ func TestBuyerStatePVC(t *testing.T) {
 		t.Errorf("litellm pod fsGroupChangePolicy = %v, want OnRootMismatch", policy)
 	}
 
-	// x402-buyer should inherit the pod-level 65532 identity and rely on
-	// fsGroup-applied local PV ownership. A container-level UID/GID 1000 is
-	// the old hostPath workaround and should not come back.
+	// x402-buyer must keep container-level UID/GID 1000 while hostPath PVs
+	// from <= v0.10.0-rc12 clusters remain in support: those PVs ignore the
+	// pod fsGroup (kubelet skips ownership management on hostPath) and hold
+	// a consumed.json written 0600 by UID 1000 — a 65532 sidecar crashloops
+	// on `load state` and takes every paid/* route down. On fresh local-type
+	// PVs the explicit UID is harmless (fsGroup 65532 grants group access).
 	containers, ok := nested(dep, "spec", "template", "spec", "containers").([]any)
 	if !ok {
 		t.Fatal("litellm Deployment has no containers")
@@ -110,11 +113,11 @@ func TestBuyerStatePVC(t *testing.T) {
 	if buyer == nil {
 		t.Fatal("x402-buyer container missing from litellm Deployment")
 	}
-	if u := nested(buyer, "securityContext", "runAsUser"); u != nil {
-		t.Errorf("x402-buyer securityContext.runAsUser = %v, want unset (inherits pod UID 65532)", u)
+	if u := nested(buyer, "securityContext", "runAsUser"); u != 1000 {
+		t.Errorf("x402-buyer securityContext.runAsUser = %v, want 1000 (legacy hostPath-PV state compat)", u)
 	}
-	if g := nested(buyer, "securityContext", "runAsGroup"); g != nil {
-		t.Errorf("x402-buyer securityContext.runAsGroup = %v, want unset (inherits pod GID 65532)", g)
+	if g := nested(buyer, "securityContext", "runAsGroup"); g != 1000 {
+		t.Errorf("x402-buyer securityContext.runAsGroup = %v, want 1000 (legacy hostPath-PV state compat)", g)
 	}
 	if nr := nested(buyer, "securityContext", "runAsNonRoot"); nr != true {
 		t.Errorf("x402-buyer securityContext.runAsNonRoot = %v, want true (restricted PSS)", nr)

internal/embed/infrastructure/base/templates/llm.yaml

@@ -176,7 +176,15 @@ spec:
       labels:
         app: litellm
       annotations:
-        configmap.reloader.stakater.com/reload: "litellm-config,x402-buyer-config,x402-buyer-auths"
+        # litellm-config only: model-list changes must roll the pod so paid
+        # routes pick up new entries. The buyer ConfigMaps are deliberately
+        # NOT listed — x402-buyer hot-reloads them via /admin/reload (driven
+        # by the serviceoffer-controller on every buy/refill), and this
+        # Deployment is strategy Recreate with one replica: reloading on
+        # x402-buyer-config/x402-buyer-auths would take the whole inference
+        # gateway down on every purchase event (CLAUDE.md pitfall 7: restart
+        # is the fallback, not the default buy path).
+        configmap.reloader.stakater.com/reload: "litellm-config"
         secret.reloader.stakater.com/reload: "litellm-secrets"
     spec:
       terminationGracePeriodSeconds: 60
@@ -304,13 +312,20 @@ spec:
           # across flow-08/11/14/13. See internal/embed/embed_image_pin_test.go.
           image: ghcr.io/obolnetwork/x402-buyer:f5d94fc@sha256:0c431eda44e9e2fe5dd50c82cf4885f9be5037e592478781c51e9c510171265c
           imagePullPolicy: IfNotPresent
-          # PSS Restricted + writable PVC. The StorageClass asks
-          # local-path-provisioner for local PVs, so kubelet applies the
-          # pod-level fsGroup to /state on mount. The sidecar inherits the
-          # pod's nonroot 65532 UID/GID and needs no root init or UID 1000
-          # alignment workaround.
+          # PSS Restricted + writable PVC. On fresh clusters the StorageClass
+          # asks local-path-provisioner for local PVs, so kubelet applies the
+          # pod-level fsGroup to /state on mount and any UID works. The
+          # explicit UID/GID 1000 below is for clusters UPGRADED in place
+          # from <= v0.10.0-rc12: their x402-buyer-state PV is hostPath-typed
+          # (kubelet ignores fsGroup there) and the old provisioner chowned
+          # the dir 1000:1000 with consumed.json written 0600 by UID 1000 —
+          # a 65532 sidecar cannot even read it and crashloops on startup
+          # (`load state` Fatalf), taking every paid/* route down. Keep 1000
+          # until hostPath PVs are out of support.
           securityContext:
             runAsNonRoot: true
+            runAsUser: 1000
+            runAsGroup: 1000
             allowPrivilegeEscalation: false
             readOnlyRootFilesystem: true
             capabilities:

internal/stack/stack_test.go

@@ -491,14 +491,22 @@ func TestLLMTemplate_IncludesPaidRouteAndBuyerSidecar(t *testing.T) {
 		`name: buyer-http`,
 		`name: x402-buyer-config`,
 		`name: x402-buyer-auths`,
-		`configmap.reloader.stakater.com/reload: "litellm-config,x402-buyer-config,x402-buyer-auths"`,
+		// litellm-config ONLY: the buyer ConfigMaps must stay out of the
+		// Reloader annotation — x402-buyer hot-reloads them via /admin/reload
+		// and the Recreate strategy would otherwise bounce the whole gateway
+		// on every buy/refill (CLAUDE.md pitfall 7).
+		`configmap.reloader.stakater.com/reload: "litellm-config"`,
 		`emptyDir:`,
 	} {
 		if !strings.Contains(out, want) {
 			t.Fatalf("llm template missing %q:\n%s", want, out)
 		}
 	}
 
+	if strings.Contains(out, `configmap.reloader.stakater.com/reload: "litellm-config,`) {
+		t.Fatal("llm template reload annotation must list litellm-config only — buyer CM writes happen per purchase and would Recreate-bounce the gateway")
+	}
+
 	if strings.Contains(out, "custom_provider_map") {
 		t.Fatalf("llm template should not require a custom provider:\n%s", out)
 	}

plans/volume-permission-hardening.md

@@ -49,6 +49,39 @@ For a fresh-stack PVC check, use only the CLI surface: create the stack, inspect
 the `local-path` StorageClass, then inspect every bound PV and confirm it has
 `.spec.local.path` and no `.spec.hostPath`.
 
+## Upgrading from <= v0.10.0-rc12 (breaking)
+
+The new model only governs PVs provisioned AFTER this change. PV specs are
+immutable: a cluster created on rc12 or earlier keeps hostPath-typed PVs,
+where the kubelet skips fsGroup ownership management entirely, and its
+hermes-data files are owned 10000:10000 (the old containerUID, chowned by the
+removed root init on every start). Running a v0.10.0 CLI `obol stack up`
+against such a cluster re-applies the new pod specs in place and the Hermes
+pod (now UID 1000, no chown init, no `fixHermesDataPVCK3dFallback`) loses
+read/write access to its own state.
+
+Supported upgrade path: **recreate the cluster**.
+
+```bash
+obol agent wallet backup            # if any agent wallet holds funds
+obol stack down && obol stack purge -f
+obol stack init && obol stack up
+obol agent wallet restore           # as needed
+```
+
+Escape hatch for clusters that must not be recreated (k3d): chown the legacy
+backing dirs to the new UID from inside the node, then restart the pods:
+
+```bash
+docker exec k3d-obol-stack-<id>-server-0 \
+  sh -c 'chown -R 1000:1000 /var/lib/rancher/k3s/storage/pvc-*hermes-data*'
+```
+
+The x402-buyer sidecar deliberately keeps container-level UID/GID 1000
+(llm.yaml) so the legacy `x402-buyer-state` PV — dir 1000:1000, consumed.json
+0600 — stays readable without any migration; do not remove that alignment
+until hostPath PVs are out of support.
+
 ## Remaining Debt
 
 OpenClaw and wallet provisioning still contain legacy host-side staging paths