merge: PR #614 — agent purchase type within hermes (rc13 fixes included)

Author: bussyjd

.agents/skills/obol-stack-dev/references/release-smoke-debugging.md

@@ -107,6 +107,21 @@ When buying from external x402 sellers (sellers running outside our k3d cluster
 - **Fix in repo**: `c2dddc1` — added module-level `USER_AGENT = os.environ.get("OBOL_BUYER_USER_AGENT", "obol-buy-x402/1.0 (+https://github.com/ObolNetwork/obol-stack)")` to `internal/embed/skills/buy-x402/scripts/buy.py`, applied in `_probe_endpoint` (kind=http), `_probe_endpoint` (kind=inference), and the paid `X-PAYMENT` request in `buy_paid_oneshot`. Tested four UAs against v1337 (`curl/*`, generic `Mozilla/*`, `Chrome/*`, custom `obol-buy-x402/*`) — all four returned 402 cleanly. The fix is "send anything that isn't `Python-urllib`", not "send a specific browser UA". Operator override: `OBOL_BUYER_USER_AGENT`.
 - **Follow-up (not yet confirmed)**: the same WAF block likely affects the Go-side controller probe at `internal/serviceoffercontroller/purchase.go:183`, since Go's `http.Client` defaults to `User-Agent: Go-http-client/1.1`. Verify against v1337 and apply the same UA override on the Go side if reproduced.
 
+### 11. "0 spent / N remaining" from the sidecar is NOT proof no debit happened
+
+The buyer sidecar's `/status` (and `PurchaseRequest.status`, and verifier logs) all report the **same local view** — they will agree with each other even when the chain disagrees with all three. rc13 mainnet OBOL self-test (2026-06-09) caught a 0.001 OBOL on-chain debit from a request that returned `HTTP 503 "Payment settlement failed"` while every signal the stack produced said "nothing was paid." The facilitator submitted the Permit2 settle tx, it mined successfully (`0xb5122d818a058e8bf529380260fa2584ba3d50bfc800f1e906faca34d3932307`), and **then** the facilitator's post-submit step returned 500.
+
+- **Fix in repo (this branch)**:
+  - Verifier preserves the facilitator's `transaction` field via `X-PAYMENT-RESPONSE` even on a non-200 `/settle` (`internal/x402/forwardauth.go` + `TestForwardAuth_SettleErrorPreservesTxHashInHeader`).
+  - Buyer sidecar treats any error response (>= 400) with `X-PAYMENT-RESPONSE.transaction != ""` as **spent on-chain**: `ConfirmSpend` the held auth, fire `OnPaymentUnsettled`, log the hash (`internal/x402/buyer/proxy.go` + `TestProxy_UpstreamErrorWithTxHash_PersistsConsume`).
+  - `buy.py` `_print_paid_request_failure` prints `⚠️  SETTLEMENT MAY HAVE COMPLETED ON-CHAIN` with the tx hash + the exact `balance --chain <X>` command when a paid call fails (>= 400) with a settle header.
+- **Not yet fixed (follow-up PRs)**:
+  - Verifier doing a receipt lookup against eRPC before returning 200 vs 5xx (would let the verifier serve the upstream response if settle landed on-chain).
+  - Settle idempotency on retry (today guarded only by Permit2 nonce reuse reverting on-chain, which burns gas).
+  - Facilitator-side: why does mainnet OBOL `/settle` return 500 *after* a successful submit? That's the hosted service (`x402.gcp.obol.tech`), not in this repo.
+- **Operator debugging recipe** (when a buyer-reported "0 spent" disagrees with a suspected debit): see `docs/observability.md` § "Verify settlement against the chain, never the sidecar snapshot" — has the exact `eth_getLogs` curl to confirm.
+- **Rule of thumb**: chain is canonical, sidecar status is a derived snapshot. The CRD itself documents this (`PurchaseRequest.status` is the controller's last reconciled snapshot, not a live counter — `CLAUDE.md` "Quick full-cycle smoke test"). For real-time auth pool state, always query the sidecar `/status`; for real-money truth, always query the chain.
+
 ## Diagnostic Patterns
 
 - **Don't confuse 503 with "verifier broken"** — almost always one of #1, #2, #5, #6, or a missing CA bundle (`paid-flows.md`).

docs/observability.md

@@ -181,6 +181,77 @@ Prometheus to answer a lifetime question, you've picked the wrong tool.
 
 ---
 
+## Verify settlement against the chain, never the sidecar snapshot
+
+The same "chain is canonical, metrics are derived" rule applies to **live
+debugging**, not just lifetime aggregates. When a paid request errors and
+the buyer reports `remaining=N, spent=0`, that is **not** evidence that no
+money moved — it is evidence that the sidecar's local counter, the
+`PurchaseRequest.status` snapshot, and the verifier's logs all agree with each
+other. The on-chain transfer event can still tell you otherwise.
+
+**This is not theoretical.** The rc13 mainnet OBOL self-test (2026-06-09)
+recorded a 0.001 OBOL on-chain debit from a request that 503'd with
+`"Payment settlement failed"`, while the buyer sidecar reported `0 spent / 2
+remaining` and the verifier logged `facilitator settle failed (500)`. The
+failure happened in the facilitator's post-submit step — the Permit2 settle
+tx had already mined. By every signal the stack gave the operator, nothing
+happened. The chain disagreed.
+
+The defenses that landed (this PR):
+
+- **Verifier**: when `facilitatorSettle` returns non-200 with a parseable body
+  that includes a `transaction` field, the tx hash is surfaced via
+  `X-PAYMENT-RESPONSE` *before* the 503 is written. Without this the on-chain
+  hash is invisible to the buyer. See `internal/x402/forwardauth.go` and
+  `TestForwardAuth_SettleErrorPreservesTxHashInHeader`.
+- **Buyer sidecar**: any error response (>= 400) with `X-PAYMENT-RESPONSE` carrying a tx hash is
+  treated as "spent on-chain" — the held auth is `ConfirmSpend`-ed (not
+  released back to the pool), `OnPaymentUnsettled` fires, and the operator
+  warning logs the hash. See `internal/x402/buyer/proxy.go` and
+  `TestProxy_UpstreamErrorWithTxHash_PersistsConsume`.
+- **buy.py CLI**: `_print_paid_request_failure` decodes the settle header on
+  any failed paid call (>= 400) and prints a loud `⚠️  SETTLEMENT MAY HAVE COMPLETED ON-CHAIN` warning
+  with the exact balance-check command.
+
+The defenses that are **deferred** (and worth flagging in any future debugging
+session):
+
+- Full receipt verification (verifier queries an RPC for the receipt status
+  before deciding 200 vs 503). The forensic fix surfaces enough for an
+  operator to reconcile manually; programmatic reconciliation is a bigger
+  plumbing change.
+- Settle idempotency on retry (today guarded only by Permit2 nonce reuse
+  reverting on-chain — that surfaces as cascading 503s, but burns gas).
+- Facilitator-side fix for the 500-after-on-chain-submit failure mode on
+  mainnet OBOL specifically. That's a hosted-service bug, not in this repo.
+
+**Debugging checklist**, when a buyer-reported "0 spent" disagrees with a
+suspected debit:
+
+```bash
+RPC=https://ethereum-rpc.publicnode.com
+blk=$(curl -s -X POST $RPC -H 'content-type: application/json' \
+  -d '{"jsonrpc":"2.0","id":1,"method":"eth_blockNumber","params":[]}' \
+  | python3 -c "import json,sys;print(int(json.load(sys.stdin)['result'],16))")
+from=$(printf '0x%x' $((blk-50000)))
+# Topic 0 = ERC-20 Transfer(address,address,uint256)
+# Topic 2 = recipient (32-byte left-padded)
+PAD=000000000000000000000000<RECIPIENT_HEX_NO_0x>
+curl -s -X POST $RPC -H 'content-type: application/json' -d "{
+  \"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"eth_getLogs\",\"params\":[{
+    \"fromBlock\":\"$from\",\"toBlock\":\"latest\",
+    \"address\":\"<TOKEN_CONTRACT>\",
+    \"topics\":[\"0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef\",null,\"0x$PAD\"]
+  }]}"
+```
+
+A `Transfer` to the expected recipient that exists while the buyer reports
+`0 spent` is the bug. Reference: rc13 mainnet OBOL tx
+[`0xb5122d818a058e8bf529380260fa2584ba3d50bfc800f1e906faca34d3932307`](https://etherscan.io/tx/0xb5122d818a058e8bf529380260fa2584ba3d50bfc800f1e906faca34d3932307).
+
+---
+
 ## Recording rule conventions
 
 Naming follows the standard Prometheus pattern:

flows/flow-16-sell-agent.sh

@@ -50,6 +50,20 @@ else
     fail "Skills mismatch: got=$got want=$expected"
 fi
 
+# §1.1.1: .no-bundled-skills marker landed on the host PVC path.
+# SeedHostFiles writes it; the cluster mounts HostHomePath into the pod via
+# the data PVC. Without the marker, Hermes' sync_skills() seeds ~24 stock
+# categories (~1 MB of SKILL.md text) into /data/.hermes/skills on every
+# launch — see internal/agentcrd/agent_contract_integration_test.go for the
+# v2026.5.28 → v2026.6.5 root cause.
+step ".no-bundled-skills marker present on host PVC"
+marker_file="$host_root/.no-bundled-skills"
+if [ -f "$marker_file" ]; then
+    pass ".no-bundled-skills marker present at $marker_file"
+else
+    fail ".no-bundled-skills marker missing at $marker_file — Hermes will re-seed bundled skills on every launch"
+fi
+
 # §1.2: Agent CR observable
 step "kubectl get agent $AGENT_NAME -n $AGENT_NS"
 phase=$("$OBOL" kubectl get agent "$AGENT_NAME" -n "$AGENT_NS" \
@@ -69,19 +83,62 @@ case "$phase" in
         ;;
 esac
 
-# §1.3: Hermes pod check
-step "Hermes pod running in $AGENT_NS"
-pod_phase=""
+# §1.3: Hermes pod check — gate on the Ready condition, not phase. Phase
+# flips to Running while containers are still booting; §1.3.1 execs into the
+# pod and would false-pass on an empty skills dir mid-boot.
+step "Hermes pod ready in $AGENT_NS"
+pod_ready=""
 for i in $(seq 1 30); do
-    pod_phase=$("$OBOL" kubectl get pods -n "$AGENT_NS" -l app.kubernetes.io/name=hermes \
-        -o jsonpath='{.items[0].status.phase}' 2>/dev/null || true)
-    [ "$pod_phase" = "Running" ] && break
+    pod_ready=$("$OBOL" kubectl get pods -n "$AGENT_NS" -l app.kubernetes.io/name=hermes \
+        -o jsonpath='{.items[0].status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || true)
+    [ "$pod_ready" = "True" ] && break
     sleep 4
 done
-if [ "$pod_phase" = "Running" ]; then
-    pass "Hermes pod Running"
+if [ "$pod_ready" = "True" ]; then
+    pass "Hermes pod Ready"
+else
+    fail "Hermes pod did not become Ready within 120s (ready=$pod_ready)"
+fi
+
+# §1.3.1: Bundled-skills contract — the marker on the host PVC was honored
+# inside the pod. This is the load-bearing CI check that should have caught
+# the v2026.5.28 bundled-skills bloat ($1 MB SKILL.md text re-seeded on every
+# launch by sync_skills(), which ignored the marker before v2026.6.5 / commit
+# 2ed96372a "blank-slate skills"). We assert the same two halves the Go
+# integration test does (internal/agentcrd/agent_contract_integration_test.go):
+#   (a) /data/.hermes/obol-skills is populated with the operator-chosen subset
+#       — proof our seeding path worked.
+#   (b) /data/.hermes/skills (the native bundled-skills dir) is absent or empty
+#       — proof Hermes honored the marker and did NOT re-seed bundled skills.
+HERMES_POD=$("$OBOL" kubectl get pods -n "$AGENT_NS" -l app.kubernetes.io/name=hermes \
+    -o jsonpath='{.items[0].metadata.name}' 2>/dev/null || true)
+
+step "Pod-side obol-skills external_dirs populated"
+if [ -n "$HERMES_POD" ]; then
+    ext_out=$("$OBOL" kubectl exec -n "$AGENT_NS" "$HERMES_POD" -c hermes -- \
+        ls -A /data/.hermes/obol-skills 2>/dev/null || true)
+    if [ -n "$(echo "$ext_out" | tr -d '[:space:]')" ]; then
+        pass "obol-skills dir in pod = $(echo "$ext_out" | tr '\n' ',' | sed 's/,$//')"
+    else
+        fail "obol-skills external_dirs is empty in pod — operator subset did not land"
+    fi
+else
+    fail "Hermes pod name not resolvable in $AGENT_NS — cannot assert pod-side skills"
+fi
+
+step "Pod-side bundled-skills dir is absent or empty"
+if [ -n "$HERMES_POD" ]; then
+    # Tolerate "No such file or directory" — that's the strongest signal the
+    # marker was honored. Any non-empty listing is a contract violation.
+    bundled_out=$("$OBOL" kubectl exec -n "$AGENT_NS" "$HERMES_POD" -c hermes -- \
+        sh -c 'ls -A /data/.hermes/skills 2>/dev/null || true' 2>/dev/null || true)
+    if [ -z "$(echo "$bundled_out" | tr -d '[:space:]')" ]; then
+        pass "/data/.hermes/skills absent/empty — bundled-skill seeding skipped"
+    else
+        fail "/data/.hermes/skills is non-empty (Hermes re-seeded bundled skills despite the marker); contents: $(echo "$bundled_out" | head -c 200)"
+    fi
 else
-    fail "Hermes pod did not reach Running within 120s (phase=$pod_phase)"
+    fail "Hermes pod name not resolvable in $AGENT_NS — cannot assert bundled-skills empty"
 fi
 
 # §1.4: Remote-signer pod (only when wallet was requested)
@@ -189,4 +246,4 @@ if [ "${FLOW_CLEANUP:-0}" = "1" ]; then
     "$OBOL" agent delete --force "$AGENT_NAME" >/dev/null 2>&1 || true
 fi
 
-summary
+emit_metrics

flows/release-smoke.sh

@@ -222,6 +222,13 @@ main() {
         "$SCRIPT_DIR/flow-10-anvil-facilitator.sh"
         "$SCRIPT_DIR/flow-08-buy.sh"
         "$SCRIPT_DIR/flow-09-lifecycle.sh"
+        # flow-16 declares a CRD sub-agent and gates it via `obol sell agent`,
+        # asserting (among other things) that the `.no-bundled-skills` marker
+        # is honored inside the Hermes pod — the contract that v2026.5.28
+        # silently broke (~100k tokens of bundled-skill bloat re-seeded on
+        # every launch, undetected because this flow wasn't in CI). Adding it
+        # here is follow-up #2 from the rc13 report PR.
+        "$SCRIPT_DIR/flow-16-sell-agent.sh"
     )
 
     for flow in "${flows[@]}"; do

internal/agentcrd/agent_contract_integration_test.go

@@ -21,8 +21,20 @@ import (
 // The unit tests in agent_test.go and serviceoffercontroller/agent_render_test.go
 // only prove that we *render* the `.no-bundled-skills` marker and the capped
 // hermes-config keys. They do NOT prove the Hermes image
-// (nousresearch/hermes-agent:v2026.5.28) actually honors them. This test closes
-// that gap end-to-end against a LIVE cluster:
+// (nousresearch/hermes-agent:v2026.6.5) actually honors them. v2026.5.28
+// shipped the marker check on the install/CLI path only; the per-launch
+// sync_skills() call ignored it and re-seeded ~24 categories from the
+// image-baked /opt/hermes/skills source on every boot, regardless of the
+// marker. v2026.6.5 (commit 2ed96372a, "blank-slate skills") added the marker
+// check to sync_skills() itself (tools/skills_sync.py:467), so the in-cluster
+// contract is honored end-to-end. This test closes that gap on a LIVE cluster.
+//
+// CI coverage: flow-16-sell-agent.sh in release-smoke asserts the same
+// pod-side bundled-skills-empty + marker-present invariants in bash. The
+// two are belt-and-braces — flow-16 catches regressions in CI, this Go test
+// is the developer-runnable white-box version. A Hermes image bump should
+// re-run both before merging (the Renovate package rule for
+// nousresearch/hermes-agent in renovate.json reminds the reviewer):
 //
 //	(1) the .no-bundled-skills marker exists on the agent's host PVC path
 //	    (agentcrd.HostNoBundledSkillsMarkerPath), so Hermes' installer/sync

internal/buy/discover.go

@@ -222,10 +222,13 @@ func PickCatalogEntry(entries []CatalogEntry, sellerURL string) (*CatalogEntry,
 		return nil, err
 	}
 
-	var inferenceOnly []CatalogEntry
+	var inferenceOnly, agentOnly []CatalogEntry
 	for _, e := range entries {
-		if strings.EqualFold(strings.TrimSpace(e.Type), "inference") {
+		switch {
+		case strings.EqualFold(strings.TrimSpace(e.Type), "inference"):
 			inferenceOnly = append(inferenceOnly, e)
+		case strings.EqualFold(strings.TrimSpace(e.Type), "agent"):
+			agentOnly = append(agentOnly, e)
 		}
 	}
 
@@ -234,6 +237,12 @@ func PickCatalogEntry(entries []CatalogEntry, sellerURL string) (*CatalogEntry,
 		for i, e := range entries {
 			if endpointPath(e.Endpoint) == wantPath {
 				if !strings.EqualFold(strings.TrimSpace(e.Type), "inference") {
+					if strings.EqualFold(strings.TrimSpace(e.Type), "agent") {
+						return nil, fmt.Errorf(
+							"seller offer %q has type=agent; `obol buy inference` only supports type=inference.\n%s",
+							e.Name, payAgentHint(e.Endpoint),
+						)
+					}
 					return nil, fmt.Errorf("seller offer %q has type=%q; obol buy inference only supports type=inference", e.Name, e.Type)
 				}
 				return &entries[i], nil
@@ -244,6 +253,18 @@ func PickCatalogEntry(entries []CatalogEntry, sellerURL string) (*CatalogEntry,
 
 	switch len(inferenceOnly) {
 	case 0:
+		// Storefront-base probe of an agent-only seller: point at pay-agent
+		// instead of the bare type=inference refusal.
+		if len(agentOnly) > 0 {
+			names := make([]string, 0, len(agentOnly))
+			for _, e := range agentOnly {
+				names = append(names, e.Name)
+			}
+			return nil, fmt.Errorf(
+				"seller advertises no inference offers (type=inference), only type=agent offers (%s).\n%s",
+				strings.Join(names, ", "), payAgentHint(agentOnly[0].Endpoint),
+			)
+		}
 		return nil, errors.New("seller advertises no inference offers (type=inference)")
 	case 1:
 		e := inferenceOnly[0]
@@ -258,6 +279,20 @@ func PickCatalogEntry(entries []CatalogEntry, sellerURL string) (*CatalogEntry,
 	}
 }
 
+// payAgentHint renders the canonical pointer for type=agent offers: they are
+// bought with the buy-x402 skill's `pay-agent` command, which streams the
+// response directly to the calling agent (memory, tool-call traces, partial
+// results) instead of pushing it behind LiteLLM as a paid alias.
+func payAgentHint(endpoint string) string {
+	return fmt.Sprintf(
+		"For type=agent offers use the buy-x402 skill's `pay-agent` command instead — it streams the response\n"+
+			"directly to the calling agent (memory, tool-call traces, partial results) without pushing it behind\n"+
+			"LiteLLM as a paid alias:\n"+
+			"  python3 ${OBOL_SKILLS_DIR:-/data/.openclaw/skills}/buy-x402/scripts/buy.py pay-agent %s --model <model> --message '<prompt>'",
+		endpoint,
+	)
+}
+
 // VerifyAgentID returns nil iff at least one of reg.Registrations matches the
 // expected ERC-8004 tokenId. The seller may publish multiple registrations
 // (one per chain); a match on any of them is sufficient.

internal/embed/embed_crd_test.go

@@ -819,7 +819,10 @@ func TestX402VerifierImage_CarriesAgentAuthFix(t *testing.T) {
 		t.Fatalf("ReadInfrastructureFile: %v", err)
 	}
 
-	const ref = "ghcr.io/obolnetwork/x402-verifier:46e63fd@sha256:a8cd7946884c9a702b5cfcfad28d1f5eac1037899303eb4e0157e3ffab7a572c"
+	// Bumped to 04bebbc (current main HEAD as of rc13) to also carry ab71481
+	// (suppress verifyOnly=false warning on the in-process settle path). The
+	// agent upstream auth fix from abfd55a remains in scope.
+	const ref = "ghcr.io/obolnetwork/x402-verifier:04bebbc@sha256:a80f72c89341a422724ad1b5d5d5da0c8cdd246b9dcabc6560e369b48ed5d775"
 	if !strings.Contains(string(data), "image: "+ref) {
 		t.Fatalf("x402-verifier image must carry agent upstream auth fix: %s", ref)
 	}
@@ -841,7 +844,7 @@ func TestServiceOfferControllerImage_CarriesSecretCreateOnlyFix(t *testing.T) {
 		t.Fatalf("ReadInfrastructureFile: %v", err)
 	}
 
-	const ref = "ghcr.io/obolnetwork/serviceoffer-controller:b39bcaa@sha256:f5afbba041f83c52c1d48c61db443138da76a12afed0bd29ba719984fc73b189"
+	const ref = "ghcr.io/obolnetwork/serviceoffer-controller:04bebbc@sha256:286d07604c001006d54a5f89ef854210ab805859c072e7b8dd89fe0c6f130d7d"
 	if !strings.Contains(string(data), "image: "+ref) {
 		t.Fatalf("serviceoffer-controller image must carry the Secret-create-only reconciler fix "+
 			"(else per-agent provisioning 403s under the no-update/patch Secret RBAC): %s", ref)