diff --git a/CLAUDE.md b/CLAUDE.md index 17219cca..132a8a13 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -9,8 +9,8 @@ Obol Stack: framework for AI agents to run decentralised infrastructure locally. ## Conventions - **Commits**: Conventional commits — `feat:`, `fix:`, `docs:`, `test:`, `chore:`, `security:` with optional scope -- **Branches**: `feat/`, `fix/`, `research/`, `codex/` prefixes -- **GitHub branch policy**: never push `codex/`-prefixed branches to GitHub from this repository; use `feat/`, `fix/`, `research/`, or another non-codex branch name before pushing +- **Branches**: `feat/`, `fix/`, `research/`, `docs/`, `chore/` prefixes +- **GitHub branch policy**: never push `codex/`-prefixed branches to GitHub from this repository; rename to `feat/`, `fix/`, `research/`, `docs/`, `chore/`, or another non-codex branch name before pushing - **Detailed architecture reference**: `@.claude/skills/obol-stack-dev/SKILL.md` (invoke with `/obol-stack-dev`) - **Review scope**: Avoid broad, vague review/delegation boundaries. State the exact files, invariants, and expected evidence before reviewing or spawning agents. Prefer concrete checks such as "controller cannot access signer/Secrets", "agent write RBAC is namespace-scoped", and "flow uses real obol CLI path" over generic "review architecture". diff --git a/docs/guides/monetize-inference.md b/docs/guides/monetize-inference.md index dc009d35..55edf910 100644 --- a/docs/guides/monetize-inference.md +++ b/docs/guides/monetize-inference.md @@ -10,28 +10,27 @@ This guide walks you through exposing a local LLM as a paid API endpoint using t > [!NOTE] > `--per-mtok` is supported for inference pricing, but phase 1 still charges an > approximate flat request price derived as `perMTok / 1000` using a fixed -> `1000 tok/request` assumption. Exact token metering is deferred to the -> follow-up `x402-meter` design described in -> [`docs/plans/per-token-metering.md`](../plans/per-token-metering.md). +> `1000 tok/request` assumption. Exact token metering is not implemented yet. > [!IMPORTANT] -> The monetize subsystem is alpha software on the `feat/secure-enclave-inference` branch. +> The monetize subsystem is alpha software. > If you encounter an issue, please open a > [GitHub issue](https://github.com/ObolNetwork/obol-stack/issues). > [!IMPORTANT] -> The current implementation is event-driven. `ServiceOffer` is the source of truth, `serviceoffer-controller` owns reconciliation, `RegistrationRequest` isolates registration side effects, and `x402-verifier` derives live routes directly from published ServiceOffers. -> Older references below to the obol-agent reconcile loop, heartbeat polling, or direct `x402-pricing` route mutation are historical. +> `ServiceOffer` is the source of truth. `serviceoffer-controller` owns +> reconciliation, `RegistrationRequest` isolates registration side effects, and +> `x402-verifier` derives live routes directly from published ServiceOffers. ## System Overview ``` SELLER (obol stack cluster) - obol sell http --> ServiceOffer CR --> Agent reconciles: + obol sell http --> ServiceOffer CR --> serviceoffer-controller reconciles: 1. ModelReady (pull model in Ollama) 2. UpstreamHealthy (health-check Ollama) - 3. PaymentGateReady (create x402 Middleware + pricing route) + 3. PaymentGateReady (create x402 Middleware) 4. RoutePublished (create HTTPRoute -> Traefik gateway) 5. Registered (ERC-8004 on-chain, optional) 6. Ready (all conditions True) @@ -177,12 +176,12 @@ That stores both values in the pricing config: - enforced phase-1 charge: `price = 0.00125 USDC / request` - approximation input: `approxTokensPerRequest = 1000` -The agent automatically reconciles the offer through six stages: +The controller automatically reconciles the offer through six stages: ``` -ModelReady [check] Agent checks /api/tags, model already cached -UpstreamHealthy [check] Agent health-checks ollama:11434 -PaymentGateReady [check] Creates Middleware x402-my-qwen + adds pricing route +ModelReady [check] Controller verifies the model is available +UpstreamHealthy [check] Controller health-checks ollama:11434 +PaymentGateReady [check] Creates Middleware x402-my-qwen RoutePublished [check] Creates HTTPRoute so-my-qwen -> ollama backend Registered -- Skipped (--register not set) Ready [check] All required conditions True @@ -191,7 +190,7 @@ Ready [check] All required conditions True Watch the progress: ```bash -# Check conditions (wait ~60s for agent heartbeat) +# Check conditions obol sell status my-qwen --namespace llm # Verify Kubernetes resources @@ -534,7 +533,7 @@ obol sell status ### Pausing -Stop serving an offer without deleting it. This removes the pricing route so requests pass through without payment: +Pause an offer without deleting it: ```bash obol sell stop my-qwen --namespace llm @@ -556,7 +555,6 @@ Deletion: - Removes the ServiceOffer CR - Cascades Middleware and HTTPRoute via OwnerReferences -- Removes the pricing route from the x402 verifier - Deactivates the ERC-8004 registration (sets `active=false`) Verify cleanup: @@ -585,7 +583,7 @@ Traefik Gateway | --> ForwardAuth to x402-verifier.x402.svc:8080 | | - | +-- Match request path against pricing routes + | +-- Match request path against published ServiceOffers | +-- No match? Return 200 (allow, free route) | +-- Match + no payment header? Return 402 + requirements | +-- Match + payment header? Verify with facilitator @@ -612,7 +610,7 @@ Traefik Gateway +--------+---------+ | +----------v-----------+ - | PaymentGateReady | (create Middleware + pricing route) + | PaymentGateReady | (create Middleware) +----------+-----------+ | +---------v----------+ @@ -630,67 +628,57 @@ Traefik Gateway ### Kubernetes Resources per ServiceOffer -When the agent reconciles a ServiceOffer named `my-qwen` in namespace `llm`: +When `serviceoffer-controller` reconciles a ServiceOffer named `my-qwen` in namespace `llm`: | Resource | Kind | Namespace | Name | |----------|------|-----------|------| | ServiceOffer | `obol.org/v1alpha1` | `llm` | `my-qwen` | | Middleware | `traefik.io/v1alpha1` | `llm` | `x402-my-qwen` | | HTTPRoute | `gateway.networking.k8s.io/v1` | `llm` | `so-my-qwen` | -| ConfigMap patch | `v1` | `x402` | `x402-pricing` (route added) | The Middleware and HTTPRoute have `ownerReferences` pointing at the ServiceOffer, so they are garbage-collected on deletion. ### Pricing Configuration -The x402 verifier reads its config from the `x402-pricing` ConfigMap: +The x402 verifier reads cluster-wide payment defaults from the +`x402-pricing` ConfigMap: ```yaml wallet: "0x70997970C51812dc3A010C7d01b50e0d17dc79C8" chain: "base-sepolia" facilitatorURL: "https://facilitator.x402.rs" verifyOnly: false -routes: - - pattern: "/services/my-qwen/*" - price: "0.001" - description: "my-qwen inference" - payTo: "0x70997970C51812dc3A010C7d01b50e0d17dc79C8" - network: "base-sepolia" ``` -This configuration is used by the `litellm-config` ConfigMap in the `llm` namespace, which LiteLLM reads for model_list configuration. - -Per-route `payTo` and `network` override the global values, enabling multiple ServiceOffers with different wallets or chains. +Published offer routes are derived from `ServiceOffer` resources rather than +being maintained manually in this ConfigMap. Per-offer `payTo` and `network` +can still override the cluster defaults. --- ## Troubleshooting -### Agent not reconciling +### Offer not reconciling -The agent reconciles on a heartbeat (~60 seconds). Check agent logs: +Check ServiceOffer conditions and controller logs: ```bash -obol kubectl logs -n openclaw-* -l app=openclaw --tail=50 +obol sell status my-qwen --namespace llm +obol kubectl logs -n x402 -l app=serviceoffer-controller --tail=50 ``` ### x402 verifier returning 200 instead of 402 -The pricing route may not have been added, or was overwritten. Check the ConfigMap: - -```bash -obol kubectl get cm x402-pricing -n x402 -o jsonpath='{.data.pricing\.yaml}' -``` - -Ensure a route matching your path exists in the `routes` list. The verifier logs its route count at startup: +The ServiceOffer may not be `Ready`, or the request path may not match the +published offer. Check the offer and the resources it owns: ```bash +obol sell status my-qwen --namespace llm +obol kubectl get middleware x402-my-qwen -n llm +obol kubectl get httproute so-my-qwen -n llm obol kubectl logs -n x402 -l app=x402-verifier --tail=10 -# Look for: "routes: 1" (or however many you expect) ``` -If routes are missing, the agent may not have reconciled yet (heartbeat is ~60s). You can also re-trigger reconciliation by deleting and re-creating the ServiceOffer. - ### Facilitator unreachable from cluster If using a self-hosted facilitator on the host, verify the k3d bridge: @@ -788,7 +776,7 @@ Replace `openclaw-obol-agent` with your actual OpenClaw namespace if different. | `obol sell http --wallet ... --chain ... --per-request ... --upstream ... --port ...` | Create a ServiceOffer | | `obol sell list` | List all ServiceOffers | | `obol sell status -n ` | Show conditions for an offer | -| `obol sell stop -n ` | Pause an offer (remove pricing route) | +| `obol sell stop -n ` | Pause an offer without deleting it | | `obol sell delete -n ` | Delete an offer and cleanup | | `obol sell status` | Show cluster pricing and registration | | `obol sell register --private-key-file ...` | Register on ERC-8004 | @@ -797,9 +785,10 @@ Replace `openclaw-obol-agent` with your actual OpenClaw namespace if different. | Resource | Namespace | Purpose | |----------|-----------|---------| -| `x402-pricing` ConfigMap | `x402` | Pricing routes and wallet config | +| `x402-pricing` ConfigMap | `x402` | Cluster-wide wallet, chain, and facilitator settings | | `x402-secrets` Secret | `x402` | Wallet address | | `x402-verifier` Deployment | `x402` | ForwardAuth payment verifier | +| `serviceoffer-controller` Deployment | `x402` | Reconciles ServiceOffers into published resources | | `serviceoffers.obol.org` CRD | (cluster) | ServiceOffer custom resource definition | | `traefik-gateway` Gateway | `traefik` | Main ingress gateway | diff --git a/docs/guides/monetize_sell_side_testing_log.md b/docs/guides/monetize_sell_side_testing_log.md deleted file mode 100644 index befd67e0..00000000 --- a/docs/guides/monetize_sell_side_testing_log.md +++ /dev/null @@ -1,399 +0,0 @@ -# Monetize Sell-Side Testing Log - -Full lifecycle walkthrough of the hardened monetize subsystem on a fresh dev cluster, using the real x402-rs facilitator against an Anvil fork of base-sepolia. - -**Branch**: `fix/review-hardening` (off `feat/secure-enclave-inference`) -**Date**: 2026-02-27 -**Cluster**: `obol-stack-sweeping-man` (k3d, 1 server node) - ---- - -## Prerequisites - -```bash -# Working directory: the obol-stack repo (or worktree) -cd /path/to/obol-stack - -# Environment — set these in every terminal session -export OBOL_DEVELOPMENT=true -export OBOL_CONFIG_DIR=$(pwd)/.workspace/config -export OBOL_BIN_DIR=$(pwd)/.workspace/bin -export OBOL_DATA_DIR=$(pwd)/.workspace/data - -# Alias for brevity (optional) -alias obol="$OBOL_BIN_DIR/obol" -``` - -**External dependencies** (must be installed separately): - -| Dependency | Install | Purpose | -|-----------|---------|---------| -| Docker | [docker.com](https://docker.com) | k3d runs inside Docker | -| Foundry (`anvil`, `cast`) | `curl -L https://foundry.paradigm.xyz \| bash && foundryup` | Local base-sepolia fork | -| Rust toolchain | [rustup.rs](https://rustup.rs) | Building x402-rs facilitator | -| Python 3 + venv | System package manager | Signing the EIP-712 payment header | -| x402-rs | `git clone https://github.com/x402-rs/x402-rs ~/Development/R&D/x402-rs` | Real x402 facilitator | -| Ollama | [ollama.com](https://ollama.com) | Local LLM inference (must be running on host) | -| `/etc/hosts` entry | `echo "127.0.0.1 obol.stack" \| sudo tee -a /etc/hosts` | `obolup.sh` does this, or add manually | - ---- - -## Phase 1: Build & Cluster - -```bash -# 1. Build the obol binary from the hardened branch -go build -o .workspace/bin/obol ./cmd/obol - -# 2. Wipe any previous cluster -obol stack down 2>/dev/null; obol stack purge -f 2>/dev/null -rm -rf "$OBOL_CONFIG_DIR" "$OBOL_DATA_DIR" - -# 3. Initialize fresh cluster config -obol stack init - -# 4. Bring up the cluster -# (builds x402-verifier Docker image locally, deploys all infrastructure) -obol stack up - -# 5. Verify — all pods should be Running -obol kubectl get pods -A -``` - -Expected: ~18 pods across namespaces (`erpc`, `kube-system`, `llm`, `monitoring`, `obol-frontend`, `openclaw-default`, `reloader`, `traefik`, `x402`). x402-verifier should have **2 replicas**. - ---- - -## Phase 2: Verify Hardening - -```bash -# Split RBAC ClusterRoles exist -obol kubectl get clusterrole openclaw-monetize-read -obol kubectl get clusterrole openclaw-monetize-workload - -# x402 namespace Role exists -obol kubectl get role openclaw-x402-pricing -n x402 - -# x402 HA: 2 replicas -obol kubectl get deploy x402-verifier -n x402 -o jsonpath='{.spec.replicas}' -# → 2 - -# PDB active -obol kubectl get pdb -n x402 -# → x402-verifier minAvailable=1 allowedDisruptions=1 -``` - ---- - -## Phase 3: Deploy Agent - -```bash -# 6. Deploy the obol-agent singleton -# - creates namespace openclaw-obol-agent -# - deploys openclaw + remote-signer pods -# - injects 24 skills (including monetize) -# - patches all 3 RBAC bindings to the agent's ServiceAccount -obol agent init - -# 7. Verify RBAC bindings point to the agent's ServiceAccount -obol kubectl get clusterrolebinding openclaw-monetize-read-binding \ - -o jsonpath='{.subjects}' -obol kubectl get clusterrolebinding openclaw-monetize-workload-binding \ - -o jsonpath='{.subjects}' -obol kubectl get rolebinding openclaw-x402-pricing-binding -n x402 \ - -o jsonpath='{.subjects}' -# All three should show: -# [{"kind":"ServiceAccount","name":"openclaw","namespace":"openclaw-obol-agent"}] -``` - ---- - -## Phase 4: Configure Payment & Create Offer - -```bash -# 8. Configure x402 pricing (seller wallet + chain) -obol sell pricing \ - --wallet 0x70997970C51812dc3A010C7d01b50e0d17dc79C8 \ - --chain base-sepolia - -# 9. Verify Ollama has the model available on the host -curl -s http://localhost:11434/api/tags | python3 -c \ - "import sys,json; [print(m['name']) for m in json.load(sys.stdin)['models']]" -# Should include qwen3:0.6b — if not: -# ollama pull qwen3:0.6b - -# 10. Create ServiceOffer CR -obol sell http my-qwen \ - --type inference \ - --model qwen3:0.6b \ - --runtime ollama \ - --per-request 0.001 \ - --network base-sepolia \ - --pay-to 0x70997970C51812dc3A010C7d01b50e0d17dc79C8 \ - --namespace llm \ - --upstream ollama \ - --port 11434 \ - --path /services/my-qwen -# → serviceoffer.obol.org/my-qwen created -``` - ---- - -## Phase 5: Agent Reconciliation - -```bash -# 11. Trigger reconciliation from inside the agent pod -# (The heartbeat cron runs every 30 min by default — -# this is the same script it would execute) -obol kubectl exec -n openclaw-obol-agent deploy/openclaw -c openclaw -- \ - python3 /data/.openclaw/skills/monetize/scripts/monetize.py process --all - -# Expected output: -# Processing 1 pending offer(s)... -# Reconciling llm/my-qwen... -# Checking if model qwen3:0.6b is available... -# Model qwen3:0.6b already available -# Health-checking http://ollama.llm.svc.cluster.local:11434/health... -# Upstream reachable (HTTP 404 — acceptable for health check) -# Creating Middleware x402-my-qwen... -# Added pricing route: /services/my-qwen/* → 0.001 USDC -# Creating HTTPRoute so-my-qwen... -# ServiceOffer llm/my-qwen is Ready - -# 12. Verify all 6 conditions are True -obol sell status my-qwen --namespace llm -# → ModelReady=True -# UpstreamHealthy=True -# PaymentGateReady=True -# RoutePublished=True -# Registered=True (Skipped) -# Ready=True -``` - ---- - -## Phase 6: Test 402 Gate (No Payment) - -```bash -# 13. Request without payment → expect HTTP 402 -curl -s -w "\nHTTP %{http_code}" -X POST \ - "http://obol.stack:8080/services/my-qwen/v1/chat/completions" \ - -H "Content-Type: application/json" \ - -d '{"model":"qwen3:0.6b","messages":[{"role":"user","content":"Hello"}],"stream":false}' - -# Expected: HTTP 402 + JSON body: -# { -# "x402Version": 1, -# "error": "Payment required for this resource", -# "accepts": [{ -# "scheme": "exact", -# "network": "base-sepolia", -# "maxAmountRequired": "1000", -# "asset": "0x036CbD53842c5426634e7929541eC2318f3dCF7e", -# "payTo": "0x70997970C51812dc3A010C7d01b50e0d17dc79C8", -# ... -# }] -# } -``` - ---- - -## Phase 7: Start x402-rs Facilitator + Anvil - -```bash -# 14. Start Anvil forking base-sepolia (background, port 8545) -anvil --fork-url https://sepolia.base.org --port 8545 --host 0.0.0.0 --silent & - -# Verify Anvil is running: -curl -s -X POST http://localhost:8545 \ - -H "Content-Type: application/json" \ - -d '{"jsonrpc":"2.0","method":"eth_chainId","params":[],"id":1}' -# → {"jsonrpc":"2.0","id":1,"result":"0x14a34"} (84532 = base-sepolia) - -# 15. Build x402-rs facilitator (first time only, ~2 min) -cd ~/Development/R\&D/x402-rs/facilitator && cargo build --release && cd - - -# 16. Start facilitator with Anvil config (background, port 4040) -# config-anvil.json points RPC at host.docker.internal:8545 -~/Development/R\&D/x402-rs/facilitator/target/release/facilitator \ - --config ~/Development/R\&D/x402-rs/config-anvil.json & - -# Verify facilitator is running: -curl -s http://localhost:4040/supported -# → {"kinds":[{"x402Version":1,"scheme":"exact","network":"base-sepolia"}, ...], -# "signers":{"eip155:84532":["0xf39Fd6e51aad88F6F4ce6aB8827279cffFb92266"]}} - -# 17. Verify buyer (Anvil account 0) has USDC on the fork -cast call 0x036CbD53842c5426634e7929541eC2318f3dCF7e \ - "balanceOf(address)(uint256)" \ - 0xf39Fd6e51aad88F6F4ce6aB8827279cffFb92266 \ - --rpc-url http://localhost:8545 -# → non-zero balance (e.g. 287787514 = ~287 USDC) -``` - ---- - -## Phase 8: Patch Verifier → Local Facilitator - -```bash -# 18. Point x402-verifier at the local x402-rs facilitator -# macOS: host.docker.internal -# Linux: host.k3d.internal -obol kubectl patch configmap x402-pricing -n x402 --type merge -p '{ - "data": { - "pricing.yaml": "wallet: 0x70997970C51812dc3A010C7d01b50e0d17dc79C8\nchain: base-sepolia\nfacilitatorURL: http://host.docker.internal:4040\nverifyOnly: false\nroutes:\n- pattern: \"/services/my-qwen/*\"\n price: \"0.001\"\n description: \"ServiceOffer my-qwen\"\n payTo: \"0x70997970C51812dc3A010C7d01b50e0d17dc79C8\"\n network: \"base-sepolia\"\n" - } -}' - -# 19. Restart verifier to pick up immediately -# (otherwise the file watcher takes 60-120s) -obol kubectl rollout restart deploy/x402-verifier -n x402 -obol kubectl rollout status deploy/x402-verifier -n x402 --timeout=60s -``` - ---- - -## Phase 9: Sign Payment & Test Paid Request - -```bash -# 20. Create venv and install eth-account -python3 -m venv /tmp/x402-venv -/tmp/x402-venv/bin/pip install eth-account --quiet - -# 21. Write the payment signing script -cat > /tmp/x402-pay.py << 'PYEOF' -#!/usr/bin/env python3 -"""Sign an x402 V1 exact payment header using Anvil account 0.""" -import json, base64, os -from eth_account import Account -from eth_account.messages import encode_typed_data - -PRIVATE_KEY = "0xac0974bec39a17e36ba4a6b4d238ff944bacb478cbed5efcae784d7bf4f2ff80" -PAYER = "0xf39Fd6e51aad88F6F4ce6aB8827279cffFb92266" -PAY_TO = "0x70997970C51812dc3A010C7d01b50e0d17dc79C8" -USDC = "0x036CbD53842c5426634e7929541eC2318f3dCF7e" -CHAIN_ID = 84532 -AMOUNT = "1000" # 0.001 USDC in 6-decimal micro-units -NONCE = "0x" + os.urandom(32).hex() - -signable = encode_typed_data(full_message={ - "types": { - "EIP712Domain": [ - {"name": "name", "type": "string"}, - {"name": "version", "type": "string"}, - {"name": "chainId", "type": "uint256"}, - {"name": "verifyingContract", "type": "address"}, - ], - "TransferWithAuthorization": [ - {"name": "from", "type": "address"}, - {"name": "to", "type": "address"}, - {"name": "value", "type": "uint256"}, - {"name": "validAfter", "type": "uint256"}, - {"name": "validBefore", "type": "uint256"}, - {"name": "nonce", "type": "bytes32"}, - ], - }, - "primaryType": "TransferWithAuthorization", - "domain": { - "name": "USDC", "version": "2", - "chainId": CHAIN_ID, "verifyingContract": USDC, - }, - "message": { - "from": PAYER, "to": PAY_TO, - "value": int(AMOUNT), - "validAfter": 0, "validBefore": 4294967295, - "nonce": bytes.fromhex(NONCE[2:]), - }, -}) - -signed = Account.sign_message(signable, PRIVATE_KEY) - -# IMPORTANT: x402-rs wire format requires validAfter/validBefore as STRINGS -payload = { - "x402Version": 1, - "scheme": "exact", - "network": "base-sepolia", - "payload": { - "signature": "0x" + signed.signature.hex(), - "authorization": { - "from": PAYER, "to": PAY_TO, - "value": AMOUNT, # string (decimal_u256) - "validAfter": "0", # string (UnixTimestamp) - "validBefore": "4294967295", # string (UnixTimestamp) - "nonce": NONCE, # string (B256 hex) - }, - }, - "resource": { - "payTo": PAY_TO, "maxAmountRequired": AMOUNT, - "asset": USDC, "network": "base-sepolia", - }, -} -print(base64.b64encode(json.dumps(payload).encode()).decode()) -PYEOF - -# 22. Generate payment header and send paid request -PAYMENT=$(/tmp/x402-venv/bin/python3 /tmp/x402-pay.py) - -curl -s -w "\nHTTP %{http_code}" -X POST \ - "http://obol.stack:8080/services/my-qwen/v1/chat/completions" \ - -H "Content-Type: application/json" \ - -H "X-PAYMENT: $PAYMENT" \ - -d '{"model":"qwen3:0.6b","messages":[{"role":"user","content":"Say hello in exactly 3 words"}],"stream":false}' - -# Expected: HTTP 200 + full Ollama inference response JSON -``` - ---- - -## Phase 10: Lifecycle Cleanup - -```bash -# 23. Stop offer (removes pricing route from ConfigMap, keeps CR) -obol sell stop my-qwen --namespace llm - -# 24. Restart verifier so removed route takes effect immediately -obol kubectl rollout restart deploy/x402-verifier -n x402 - -# 25. Verify endpoint is now free (no payment required) -curl -s -w "\nHTTP %{http_code}" -X POST \ - "http://obol.stack:8080/services/my-qwen/v1/chat/completions" \ - -H "Content-Type: application/json" \ - -d '{"model":"qwen3:0.6b","messages":[{"role":"user","content":"Hello"}],"stream":false}' -# → HTTP 200 (free endpoint, no 402) - -# 26. Full delete — removes CR + Middleware + HTTPRoute (ownerRef cascade) -obol sell delete my-qwen --namespace llm --force - -# 27. Verify everything is cleaned up -obol kubectl get serviceoffers,middleware,httproutes -n llm -# → No resources found in llm namespace. - -# 28. Stop background processes and clean up temp files -pkill -f "anvil.*fork-url" -pkill -f "facilitator.*config-anvil" -rm -rf /tmp/x402-venv /tmp/x402-pay.py -``` - ---- - -## Reference: Key Addresses - -| Role | Address | Note | -|------|---------|------| -| Seller (payTo) | `0x70997970C51812dc3A010C7d01b50e0d17dc79C8` | Anvil account 1 | -| Buyer (payer) | `0xf39Fd6e51aad88F6F4ce6aB8827279cffFb92266` | Anvil account 0 | -| Buyer private key | `0xac0974bec39a17e36ba4a6b4d238ff944bacb478cbed5efcae784d7bf4f2ff80` | Anvil default — never use in production | -| USDC (base-sepolia) | `0x036CbD53842c5426634e7929541eC2318f3dCF7e` | Circle USDC on base-sepolia | -| Chain ID | `84532` | base-sepolia | - -## Reference: Key Gotchas - -| Gotcha | Detail | -|--------|--------| -| **macOS vs Linux host bridging** | macOS: `host.docker.internal`. Linux: `host.k3d.internal` (step 18) | -| **x402-rs timestamp format** | `validAfter`/`validBefore` must be **strings** (`"0"`, `"4294967295"`), not integers. x402-rs `UnixTimestamp` deserializes from stringified u64 | -| **ConfigMap propagation delay** | x402-verifier file watcher takes 60-120s. Use `kubectl rollout restart` for immediate effect | -| **Heartbeat interval** | 30 minutes by default. For interactive testing, exec into the pod and run `monetize.py process --all` manually (step 11) | -| **`/etc/hosts`** | Must have `127.0.0.1 obol.stack`. `obolup.sh` sets this during install, or add manually | -| **`OBOL_DEVELOPMENT=true`** | Required for `obol stack up` to build the x402-verifier Docker image locally instead of pulling from registry | -| **Anvil fork freshness** | Each `anvil` restart creates a fresh fork. USDC balances come from the forked base-sepolia state at the time of fork | -| **x402-rs `config-anvil.json`** | Ships with the x402-rs repo. Points `eip155:84532` RPC at `host.docker.internal:8545` (Anvil). Adjust if your Anvil is on a different port | diff --git a/docs/guides/monetize_test_coverage_report.md b/docs/guides/monetize_test_coverage_report.md deleted file mode 100644 index d4c0262b..00000000 --- a/docs/guides/monetize_test_coverage_report.md +++ /dev/null @@ -1,666 +0,0 @@ -# Monetize Subsystem — Test Coverage Report - -**Branch**: `fix/review-hardening` (off `feat/secure-enclave-inference`) -**Date**: 2026-02-27 -**Total integration tests**: 46 across 3 files - ---- - -## Section Overview - -``` -┌──────────────────────────────────────────────────────────────────┐ -│ TEST PYRAMID │ -│ │ -│ ▲ │ -│ ╱ ╲ Phase 8: FULL (1) │ -│ ╱ ╲ ← tunnel+Ollama+x402-rs+EIP-712 │ -│ ╱─────╲ │ -│ ╱ ╲ Phase 5+: Real Facilitator (1) │ -│ ╱ ╲ ← real x402-rs, real EIP-712 │ -│ ╱───────────╲ │ -│ ╱ ╲ Phase 6+7: Tunnel + Fork (5) │ -│ ╱ ╲ ← real Ollama, mock facilitator │ -│ ╱─────────────────╲ │ -│ ╱ ╲ Phase 4+5: Payment + E2E (8) │ -│ ╱ ╲ ← mock facilitator, real gate │ -│ ╱─────────────────╲ │ -│ ╱ ╲ Phase 3: Routing (6) │ -│ ╱ ╲ ← real Traefik, Anvil RPC │ -│ ╱───────────────────────╲ │ -│ ╱ ╲ Phase 2: RBAC + Recon (6) │ -│ ╱ ╲ ← real agent in pod │ -│ ╱─────────────────────────────╲ │ -│ ╱ ╲ Phase 1: CRD (7) │ -│ ╱ ╲ ← schema validation │ -│ ╱───────────────────────────────────╲ │ -│ ╱ ╲ Base: Inference (12)│ -│ ╱_______________________________________╲ ← Ollama + skills │ -│ │ -└──────────────────────────────────────────────────────────────────┘ -``` - ---- - -## Phase 1 — CRD Lifecycle (7 tests) - -**What it covers**: ServiceOffer custom resource schema validation, CRUD operations, printer columns, status subresource isolation. - -**Realism**: Low (data-plane only, no reconciliation or traffic). - -``` -┌─────────────────────────────────────────────────────┐ -│ TEST BOUNDARY │ -│ │ -│ kubectl apply ──▶ ┌──────────────────┐ │ -│ │ ServiceOffer CR │ │ -│ kubectl get ──▶ │ (obol.org CRD) │ │ -│ └──────────────────┘ │ -│ kubectl patch ──▶ │ │ -│ kubectl delete──▶ ▼ │ -│ API Server validates: │ -│ ✓ wallet regex (^0x[0-9a-fA-F]{40}$)│ -│ ✓ status subresource isolation │ -│ ✓ printer columns (TYPE, PRICE) │ -│ │ -│ ┌─────────────────────────────────────────────┐ │ -│ │ NOT TESTED: reconciler, routing, payment │ │ -│ └─────────────────────────────────────────────┘ │ -└─────────────────────────────────────────────────────┘ -``` - -| Test | What It Proves | -|------|----------------| -| `CRD_Exists` | CRD installed in cluster | -| `CRD_CreateGet` | Spec fields round-trip correctly | -| `CRD_List` | kubectl list works | -| `CRD_StatusSubresource` | Status patch doesn't mutate spec | -| `CRD_WalletValidation` | Invalid wallet rejected by API server | -| `CRD_PrinterColumns` | `kubectl get` shows TYPE, PRICE, NETWORK | -| `CRD_Delete` | CR deletion works | - -**Gap vs real world**: No agent involvement. A real user runs `obol sell http`, not raw kubectl. - ---- - -## Phase 2 — RBAC + Reconciliation (6 tests) - -**What it covers**: Split RBAC roles exist and are bound, agent can read/write CRs from inside pod, reconciler handles unhealthy upstreams, idempotent re-processing. - -**Realism**: Medium (real agent pod, real RBAC, but no traffic or payment). - -``` -┌─────────────────────────────────────────────────────────────────┐ -│ TEST BOUNDARY │ -│ │ -│ ┌─────────────┐ RBAC Check ┌─────────────────────────┐ │ -│ │ Test Runner │ ────────────────▶ │ ClusterRole: │ │ -│ │ (kubectl get)│ │ openclaw-monetize-read │ │ -│ └─────────────┘ │ openclaw-monetize-wkld │ │ -│ │ │ Role: │ │ -│ │ │ openclaw-x402-pricing │ │ -│ │ └─────────────────────────┘ │ -│ │ │ -│ │ kubectl exec │ -│ ▼ │ -│ ┌─────────────────────────────────┐ │ -│ │ obol-agent pod │ │ -│ │ monetize.py process │──▶ ServiceOffer CR │ -│ │ monetize.py process --all │ (status conditions) │ -│ │ monetize.py list │ │ -│ └─────────────────────────────────┘ │ -│ │ │ -│ ▼ │ -│ UpstreamHealthy=False (no real upstream) │ -│ HEARTBEAT_OK (no pending offers) │ -│ │ -│ ┌──────────────────────────────────────────────────────────┐ │ -│ │ NOT TESTED: Traefik routing, x402 gate, payment, tunnel │ │ -│ └──────────────────────────────────────────────────────────┘ │ -└─────────────────────────────────────────────────────────────────┘ -``` - -| Test | What It Proves | -|------|----------------| -| `RBAC_ClusterRolesExist` | Split RBAC roles deployed by k3s manifests | -| `RBAC_BindingsPatched` | `obol agent init` patches all 3 bindings | -| `Monetize_ListEmpty` | Agent skill lists zero offers | -| `Monetize_ProcessAllEmpty` | Heartbeat returns OK with no work | -| `Monetize_ProcessUnhealthy` | Sets UpstreamHealthy=False for missing svc | -| `Monetize_Idempotent` | Second reconcile doesn't error | - -**Gap vs real world**: No upstream service exists. Reconciliation never reaches PaymentGateReady or RoutePublished. - ---- - -## Phase 3 — Routing with Anvil Upstream (6 tests) - -**What it covers**: Full 6-condition reconciliation with a real upstream (Anvil fork), Traefik Middleware + HTTPRoute creation, traffic forwarding, owner-reference cascade on delete. - -**Realism**: Medium-High (real cluster networking, real Traefik, real upstream). No payment gate yet. - -``` -┌─────────────────────────────────────────────────────────────────────┐ -│ TEST BOUNDARY │ -│ │ -│ ┌──────────┐ │ -│ │ Anvil │ ◀── Host machine (port N) │ -│ │ (fork of │ forking Base Sepolia │ -│ │ base-sep)│ │ -│ └────┬─────┘ │ -│ │ ClusterIP + EndpointSlice │ -│ │ (anvil-rpc.test-ns.svc) │ -│ ▼ │ -│ ┌──────────────────────────────────────────────────────────┐ │ -│ │ k3d cluster │ │ -│ │ │ │ -│ │ Agent reconciles: │ │ -│ │ ✓ UpstreamHealthy (HTTP health-check to Anvil) │ │ -│ │ ✓ PaymentGateReady (Middleware created) │ │ -│ │ ✓ RoutePublished (HTTPRoute created) │ │ -│ │ ✓ Ready │ │ -│ │ │ │ -│ │ ┌─────────────┐ ┌──────────────┐ ┌──────────┐ │ │ -│ │ │ Traefik GW │────▶│ HTTPRoute │────▶│ Anvil │ │ │ -│ │ │ :8080 │ │ /services/x │ │ upstream │ │ │ -│ │ └─────────────┘ └──────────────┘ └──────────┘ │ │ -│ │ │ │ -│ │ curl POST obol.stack:8080/services/x │ │ -│ │ → eth_blockNumber response from Anvil ✓ │ │ -│ └──────────────────────────────────────────────────────────┘ │ -│ │ -│ ┌────────────────────────────────────────────────────────────┐ │ -│ │ NOT TESTED: x402 ForwardAuth (no facilitator), no 402 │ │ -│ └────────────────────────────────────────────────────────────┘ │ -└─────────────────────────────────────────────────────────────────────┘ -``` - -| Test | What It Proves | -|------|----------------| -| `Route_AnvilUpstream` | Anvil responds locally | -| `Route_FullReconcile` | All 4 conditions reach True | -| `Route_MiddlewareCreated` | ForwardAuth Middleware exists | -| `Route_HTTPRouteCreated` | HTTPRoute has correct parentRef | -| `Route_TrafficRoutes` | HTTP through Traefik reaches Anvil | -| `Route_DeleteCascades` | ownerRef GC cleans up derived resources | - -**Gap vs real world**: No payment gate. Requests go straight through without x402 gating. Free endpoint, not monetized. - ---- - -## Phase 4 — Payment Gate (4 tests) - -**What it covers**: x402-verifier health, 402 response without payment, 402 response body format (x402 spec compliance), 200 response with mock payment. - -**Realism**: Medium-High. Real x402-verifier, real Traefik ForwardAuth. Mock facilitator always says `isValid: true`. - -``` -┌──────────────────────────────────────────────────────────────────────┐ -│ TEST BOUNDARY │ -│ │ -│ ┌───────┐ POST /services/x ┌──────────┐ ForwardAuth │ -│ │Client │ ─────────────────────▶ │ Traefik │ ──────────────▶ │ -│ │(test) │ │ Gateway │ │ │ -│ └───────┘ └──────────┘ │ │ -│ │ │ ▼ │ -│ │ │ ┌──────────────┐│ -│ │ No X-PAYMENT header │ │ x402-verifier││ -│ │ ──────────────────▶ │ │ (real pod) ││ -│ │ │ │ ││ -│ │ ◀── 402 + pricing JSON │ │ Checks: ││ -│ │ │ │ ✓ route match││ -│ │ │ │ ✓ has header ││ -│ │ X-PAYMENT: │ │ ✓ call facil.││ -│ │ ──────────────────▶ │ │ ││ -│ │ │ │ ┌────────┐ ││ -│ │ │ │ │ Mock │ ││ -│ │ ◀── 200 + Anvil response │ │ │ Facil. │ ││ -│ │ │ │ │ always │ ││ -│ │ │ │ │ valid │ ││ -│ │ │ │ └────────┘ ││ -│ │ │ └──────────────┘│ -│ │ -│ ┌──────────────────────────────────────────────────────────────┐ │ -│ │ MOCK: facilitator (no real signature validation) │ │ -│ │ MOCK: payment header (fake JSON, not real EIP-712) │ │ -│ └──────────────────────────────────────────────────────────────┘ │ -└──────────────────────────────────────────────────────────────────────┘ -``` - -| Test | What It Proves | -|------|----------------| -| `PaymentGate_VerifierHealthy` | /healthz and /readyz return 200 | -| `PaymentGate_402WithoutPayment` | No payment → 402 | -| `PaymentGate_RequirementsFormat` | 402 body matches x402 spec | -| `PaymentGate_200WithPayment` | Mock payment → 200 | - -**Gap vs real world**: The facilitator never validates the EIP-712 signature. Any well-formed JSON base64 header passes. Wire format bugs (string vs int types) are invisible. - ---- - -## Phase 5 — Full E2E CLI-Driven (3 tests) - -**What it covers**: `obol sell http` CLI → CR creation → agent reconciliation → 402 → 200 → `obol sell list/status/delete`. Heartbeat auto-reconciliation (90s wait). - -**Realism**: High for the CLI path. Still uses mock facilitator for payment. - -``` -┌──────────────────────────────────────────────────────────────────────┐ -│ TEST BOUNDARY │ -│ │ -│ ┌────────────────┐ │ -│ │ obol sell│ │ -│ │ offer my-qwen │ ──▶ ServiceOffer CR │ -│ │ --type inference │ │ -│ │ --model qwen3 │ │ -│ │ --per-request .. │ │ -│ └────────────────┘ │ -│ │ │ -│ ▼ │ -│ ┌──────────────────────────────────────────────────────────────────┐ │ -│ │ Agent pod (autonomous reconciliation) │ │ -│ │ │ │ -│ │ monetize.py process ──▶ 6 conditions ──▶ Ready=True │ │ -│ │ │ │ -│ │ OR: heartbeat cron (every 30min) auto-reconciles │ │ -│ └──────────────────────────────────────────────────────────────────┘ │ -│ │ │ -│ ▼ │ -│ ┌──────────────────────────────────────────────────────────────────┐ │ -│ │ obol sell list → shows offer │ │ -│ │ obol sell status → shows all conditions │ │ -│ │ obol sell delete → cleans up CR + derived resources │ │ -│ └──────────────────────────────────────────────────────────────────┘ │ -│ │ -│ Still uses mock facilitator for payment verification. │ -└──────────────────────────────────────────────────────────────────────┘ -``` - -| Test | What It Proves | -|------|----------------| -| `E2E_OfferLifecycle` | Full CLI → create → reconcile → pay → delete | -| `E2E_HeartbeatReconciles` | Cron-driven reconciliation without manual trigger | -| `E2E_ListAndStatus` | CLI query commands work | - -**Gap vs real world**: Mock facilitator. No real model (Anvil upstream, not Ollama). - ---- - -## Phase 6 — Tunnel E2E + Ollama (2 tests) - -**What it covers**: Real Ollama inference through the full stack, including Cloudflare tunnel accessibility. Agent-autonomous offer management. - -**Realism**: Very High for the local path. Tunnel tests require CF credentials. - -``` -┌───────────────────────────────────────────────────────────────────────────┐ -│ TEST BOUNDARY │ -│ │ -│ ┌─────────┐ POST /services/x/v1/chat/completions │ -│ │ Client │ ────────────────────────────────────────▶ │ -│ └─────────┘ │ │ -│ │ ▼ │ -│ │ ┌──────────┐ ForwardAuth ┌──────────────────┐ │ -│ │ │ Traefik │ ──────────────▶ │ x402-verifier │ │ -│ │ │ Gateway │ │ → mock facilitator│ │ -│ │ └──────────┘ └──────────────────┘ │ -│ │ │ │ -│ │ │ payment valid │ -│ │ ▼ │ -│ │ ┌──────────┐ │ -│ │ │ Ollama │ ← REAL model (qwen3:0.6b) │ -│ │ │ (llm ns) │ REAL inference response │ -│ │ └──────────┘ │ -│ │ │ -│ │ Also tests via tunnel: │ -│ │ ┌─────────────────────┐ │ -│ │ │ Cloudflare Tunnel │ ← if CF credentials configured │ -│ │ │ https:// │ │ -│ │ └─────────────────────┘ │ -│ │ │ -│ ┌────────────────────────────────────────────────────────────────┐ │ -│ │ REAL: Ollama inference, Traefik routing, x402-verifier │ │ -│ │ MOCK: facilitator (still always-valid) │ │ -│ │ OPTIONAL: CF tunnel (skipped without credentials) │ │ -│ └────────────────────────────────────────────────────────────────┘ │ -└───────────────────────────────────────────────────────────────────────────┘ -``` - -| Test | What It Proves | -|------|----------------| -| `Tunnel_OllamaMonetized` | Real model → real inference → mock payment → response | -| `Tunnel_AgentAutonomousMonetize` | Agent creates/manages offer without CLI | - -**Gap vs real world**: Mock facilitator. Real-world buyers send real EIP-712 signatures. - ---- - -## Phase 7 — Fork Validation with Mock Facilitator (2 tests) - -**What it covers**: Anvil-fork-backed upstream with mock facilitator verify/settle tracking, agent error recovery from bad upstream state. - -**Realism**: Medium-High. Real on-chain environment (forked), but fake payment validation. - -``` -┌──────────────────────────────────────────────────────────────────────┐ -│ TEST BOUNDARY │ -│ │ -│ ┌──────────┐ ┌─────────────────┐ │ -│ │ Anvil │ ◀── fork of Base Sepolia │ Mock Facilitator│ │ -│ │ (real │ real block numbers │ ✓ /verify │ │ -│ │ chain │ real chain ID 84532 │ → always valid│ │ -│ │ state) │ │ ✓ /settle │ │ -│ └──────────┘ │ → always ok │ │ -│ │ │ Tracks call │ │ -│ │ EndpointSlice │ counts only │ │ -│ ▼ └─────────────────┘ │ -│ ┌───────────────────────────────────┐ │ │ -│ │ Full reconciliation pipeline │ │ │ -│ │ ✓ UpstreamHealthy (Anvil health) │ │ │ -│ │ ✓ PaymentGateReady │ │ │ -│ │ ✓ RoutePublished │ │ │ -│ │ ✓ Ready │◀───────────┘ │ -│ │ │ │ -│ │ Also tests: │ │ -│ │ ✓ Pricing route in ConfigMap │ │ -│ │ ✓ Delete cleans up pricing route │ │ -│ │ ✓ Agent self-heals from bad state │ │ -│ └───────────────────────────────────┘ │ -│ │ -│ ┌──────────────────────────────────────────────────────────────┐ │ -│ │ MOCK: facilitator (no signature validation, no USDC check) │ │ -│ │ MOCK: payment header (fake JSON blob) │ │ -│ └──────────────────────────────────────────────────────────────┘ │ -└──────────────────────────────────────────────────────────────────────┘ -``` - -| Test | What It Proves | -|------|----------------| -| `Fork_FullPaymentFlow` | 402 → 200 with mock, verify/settle called | -| `Fork_AgentSkillIteration` | Agent recovers from unreachable upstream | - -**Gap vs real world**: Facilitator never validates signatures. USDC balance irrelevant. - ---- - -## Phase 5+ — Real Facilitator Payment (1 test) ← CLOSEST TO PRODUCTION - -**What it covers**: The entire payment cryptography stack. Real x402-rs facilitator binary, real EIP-712 TransferWithAuthorization signatures, real USDC balance on Anvil fork, real signature validation. - -**Realism**: Very High. The only mock remaining is the chain settlement (Anvil resets after test). - -``` -┌──────────────────────────────────────────────────────────────────────────┐ -│ TEST BOUNDARY │ -│ │ -│ ┌──────────┐ Buyer: Anvil Account[0] │ -│ │ go test │ 10 USDC minted via anvil_setStorageAt │ -│ │ │ │ -│ │ Signs real EIP-712 │ -│ │ TransferWithAuthorization │ -│ │ (ERC-3009) │ -│ │ │ -│ │ ┌─────────────────────────────────────┐ │ -│ │ │ TypedData: │ │ -│ │ │ domain: USD Coin / v2 / 84532 │ │ -│ │ │ from: buyer address │ │ -│ │ │ to: seller address │ │ -│ │ │ value: "1000" (0.001 USDC) │ │ -│ │ │ validAfter: "0" ← STRING! │ │ -│ │ │ validBefore: "4294967295" ← STRING│ │ -│ │ │ nonce: random 32 bytes │ │ -│ │ └─────────────────────────────────────┘ │ -│ └──────────┘ │ -│ │ │ -│ │ X-PAYMENT: base64(envelope) │ -│ ▼ │ -│ ┌──────────┐ ForwardAuth ┌──────────────────┐ │ -│ │ Traefik │ ───────────────▶ │ x402-verifier │ │ -│ │ Gateway │ │ (real pod) │ │ -│ └──────────┘ └────────┬─────────┘ │ -│ │ │ │ -│ │ │ POST /verify │ -│ │ ▼ │ -│ │ ┌──────────────────┐ │ -│ │ │ x402-rs │ ← REAL binary │ -│ │ │ facilitator │ │ -│ │ │ │ │ -│ │ │ ✓ Decodes header │ │ -│ │ │ ✓ Validates EIP │ │ -│ │ │ 712 signature │ │ -│ │ │ ✓ Checks USDC │ │ -│ │ │ balance on │ │ -│ │ │ Anvil fork │ │ -│ │ │ ✓ Returns │ │ -│ │ │ isValid: true │ │ -│ │ └────────┬─────────┘ │ -│ │ │ │ -│ │ │ connected to: │ -│ │ ▼ │ -│ │ ┌──────────────────┐ │ -│ │ │ Anvil Fork │ ← REAL chain state │ -│ │ │ (Base Sepolia) │ │ -│ │ │ chain ID: 84532 │ │ -│ │ │ │ │ -│ │ │ Has USDC balance │ │ -│ │ │ for buyer address │ │ -│ │ └──────────────────┘ │ -│ │ │ -│ │ 200 OK │ -│ ▼ │ -│ Response from Anvil (eth_blockNumber) │ -│ │ -│ ┌───────────────────────────────────────────────────────────────────┐ │ -│ │ REAL: x402-rs binary, EIP-712 signing, USDC state, verifier, │ │ -│ │ Traefik ForwardAuth, agent reconciliation, CRD lifecycle │ │ -│ │ SIMULATED: chain (Anvil fork, not mainnet), settlement (no │ │ -│ │ actual USDC transfer, Anvil state resets) │ │ -│ └───────────────────────────────────────────────────────────────────┘ │ -└──────────────────────────────────────────────────────────────────────────┘ -``` - -| Test | What It Proves | -|------|----------------| -| `Fork_RealFacilitatorPayment` | Real EIP-712 → real x402-rs → real validation → 200 | - -**Gap vs real world**: Settlement doesn't transfer real USDC (Anvil fork resets). No real L1/L2 block confirmation. No Cloudflare tunnel in this test. - ---- - -## Phase 8 — Full Stack: Tunnel + Ollama + Real Facilitator (1 test) ← PRODUCTION EQUIVALENT - -**What it covers**: Everything. Real Ollama inference, real x402-rs facilitator, real EIP-712 signatures, USDC-funded Anvil fork, and requests entering through the Cloudflare quick tunnel's dynamic `*.trycloudflare.com` URL. - -**Realism**: Maximum. This is a production sell-side scenario with the only difference being Anvil (not mainnet) and a quick tunnel (not a persistent named tunnel). - -``` -┌──────────────────────────────────────────────────────────────────────────────┐ -│ TEST BOUNDARY │ -│ │ -│ BUYER (test runner) │ -│ ┌──────────────────────────────────────────────────────────────────────┐ │ -│ │ 1. Signs real EIP-712 TransferWithAuthorization (ERC-3009) │ │ -│ │ domain: USD Coin / v2 / 84532 │ │ -│ │ from: 0xf39F... (Anvil account[0], funded with 10 USDC) │ │ -│ │ to: 0x7099... (seller) │ │ -│ │ value: "1000" (0.001 USDC) │ │ -│ │ nonce: random 32 bytes │ │ -│ └──────────────────────────────────────────────────────────────────────┘ │ -│ │ │ -│ │ POST https://.trycloudflare.com/services/test-tunnel-real/ │ -│ │ /v1/chat/completions │ -│ │ X-PAYMENT: base64(real EIP-712 envelope) │ -│ ▼ │ -│ ┌──────────────────────────────────────┐ │ -│ │ Cloudflare Edge (quick tunnel) │ ← REAL Cloudflare infrastructure │ -│ │ *.trycloudflare.com │ dynamic URL, non-persistent │ -│ │ TLS termination │ │ -│ └────────────────┬─────────────────────┘ │ -│ │ cloudflared connector (k3d pod) │ -│ ▼ │ -│ ┌──────────────────────────────────────┐ │ -│ │ Traefik Gateway (:443 internal) │ ← REAL Traefik, Gateway API │ -│ │ HTTPRoute: /services/test-tunnel-* │ │ -│ │ ForwardAuth middleware │ │ -│ └────────────────┬─────────────────────┘ │ -│ │ ForwardAuth request │ -│ ▼ │ -│ ┌──────────────────────────────────────┐ │ -│ │ x402-verifier (2 replicas, PDB) │ ← REAL verifier pod │ -│ │ Extracts X-PAYMENT header │ │ -│ │ Looks up pricing route in ConfigMap │ │ -│ │ Calls facilitator /verify │ │ -│ └────────────────┬─────────────────────┘ │ -│ │ POST /verify │ -│ ▼ │ -│ ┌──────────────────────────────────────┐ │ -│ │ x402-rs facilitator (host process) │ ← REAL Rust binary │ -│ │ │ │ -│ │ ✓ Decodes x402 V1 envelope │ │ -│ │ ✓ Recovers signer from EIP-712 sig │ │ -│ │ ✓ Checks USDC balance on Anvil │ │ -│ │ ✓ Validates nonce not replayed │ │ -│ │ ✓ Returns isValid: true + payer │ │ -│ └────────────────┬─────────────────────┘ │ -│ │ connected to: │ -│ ▼ │ -│ ┌──────────────────────────────────────┐ │ -│ │ Anvil Fork (host process) │ ← REAL chain state (Base Sepolia) │ -│ │ chain ID: 84532 │ USDC balances, nonce tracking │ -│ │ 10 USDC minted to buyer │ │ -│ └──────────────────────────────────────┘ │ -│ │ -│ ◀── verifier returns 200 (payment valid) │ -│ │ │ -│ ▼ Traefik forwards to upstream │ -│ ┌──────────────────────────────────────┐ │ -│ │ Ollama (llm namespace) │ ← REAL model inference │ -│ │ model: qwen2.5 / qwen3:0.6b │ actual LLM generation │ -│ │ │ │ -│ │ POST /v1/chat/completions │ │ -│ │ → "say hello in one word" │ │ -│ │ ← {"choices":[{"message":...}]} │ │ -│ └──────────────────────────────────────┘ │ -│ │ -│ ◀── 200 + inference response returned to buyer via tunnel │ -│ │ -│ ┌───────────────────────────────────────────────────────────────────────┐ │ -│ │ REAL: tunnel, Traefik, x402-verifier, x402-rs, EIP-712, USDC, │ │ -│ │ Ollama, agent reconciliation, CRD, RBAC, Gateway API │ │ -│ │ SIMULATED: chain (Anvil fork, not mainnet), settlement │ │ -│ │ NOT PERSISTENT: quick tunnel URL changes on restart │ │ -│ └───────────────────────────────────────────────────────────────────────┘ │ -└──────────────────────────────────────────────────────────────────────────────┘ -``` - -| Test | What It Proves | -|------|----------------| -| `Tunnel_RealFacilitatorOllama` | Buyer → CF tunnel → x402 gate → real EIP-712 validation → real Ollama inference → response via tunnel | - -**What makes this different from every other test**: - -| Component | Phase 6 (existing) | Phase 5+ (Anvil) | Phase 8 (this) | -|-----------|-------------------|-------------------|----------------| -| Inference | Real Ollama | Anvil RPC | Real Ollama | -| Facilitator | Mock (always valid) | Real x402-rs | Real x402-rs | -| Payment signature | Fake JSON blob | Real EIP-712 | Real EIP-712 | -| USDC balance | N/A | Minted on Anvil | Minted on Anvil | -| Entry point | obol.stack:8080 | obol.stack:8080 | **\*.trycloudflare.com** | -| TLS | None (HTTP) | None (HTTP) | **Real TLS** (CF edge) | - -**Gap vs real world**: Quick tunnel URL is ephemeral (not a persistent `myagent.example.com`). USDC settlement doesn't transfer real tokens (Anvil resets). No real L1/L2 block finality. - ---- - -## Base Tests — Inference + Skills (12 tests) - -**What they cover**: Ollama/Anthropic/OpenAI/Google/Zhipu inference through LiteLLM, skill staging and injection, skill visibility in pod, skill-driven agent responses. - -**Realism**: Very High for inference path. These are the "does the AI actually work" tests. - -Not directly part of the monetize subsystem, but they validate the upstream service that gets monetized. - ---- - -## Realism Comparison Matrix - -``` - CRD RBAC Agent Traefik x402 Facil. EIP-712 USDC Ollama Tunnel TLS - ─── ──── ───── ─────── ──── ────── ─────── ──── ────── ────── ─── -Phase 1 (CRD) ✓ -Phase 2 (RBAC) ✓ ✓ ✓ -Phase 3 (Route) ✓ ✓ ✓ ✓ -Phase 4 (Gate) ✓ ✓ ✓ ✓ ✓ MOCK MOCK -Phase 5 (E2E) ✓ ✓ ✓ ✓ ✓ MOCK MOCK -Phase 6 (Tunnel) ✓ ✓ ✓ ✓ ✓ MOCK MOCK ✓ ✓ ✓ -Phase 7 (Fork) ✓ ✓ ✓ ✓ ✓ MOCK MOCK N/A -Phase 5+ (Real) ✓ ✓ ✓ ✓ ✓ REAL REAL REAL -Phase 8 (FULL) ✓ ✓ ✓ ✓ ✓ REAL REAL REAL ✓ ✓ ✓ - - ✓ = real component MOCK = simulated REAL = production-equivalent -``` - ---- - -## What's Still Not Tested - -| Gap | Impact | Mitigation | -|-----|--------|------------| -| **Real USDC settlement** | Anvil fork doesn't persist transfers | Would need Base Sepolia testnet with real USDC faucet | -| **Persistent named tunnel** | Quick tunnel URL is ephemeral | Phase 8 uses quick tunnel; persistent requires `obol tunnel provision` with CF credentials | -| **Concurrent buyers** | All tests are single-buyer | Add load test with multiple signed payments | -| **ERC-8004 registration** | `obol sell register` not tested end-to-end | Would need real Base Sepolia tx (gas costs) | -| **Price change hot-reload** | Agent updates price in CR → verifier picks up new amount | Test exists partially in Phase 4 format checks | -| **Buy-side flow** | No buyer CLI/SDK test | Planned as next phase | - ---- - -## Running the Tests - -```bash -# Prerequisites -export OBOL_DEVELOPMENT=true -export OBOL_CONFIG_DIR=$(pwd)/../../.workspace/config -export OBOL_BIN_DIR=$(pwd)/../../.workspace/bin -export OBOL_DATA_DIR=$(pwd)/../../.workspace/data - -# Phase 1-3: CRD + RBAC + Routing (fast, ~2min) -go test -tags integration -v -timeout 5m \ - -run 'TestIntegration_CRD_|TestIntegration_RBAC_|TestIntegration_Monetize_|TestIntegration_Route_' \ - ./internal/openclaw/ - -# Phase 4-5: Payment gate + E2E (medium, ~5min) -go test -tags integration -v -timeout 10m \ - -run 'TestIntegration_PaymentGate_|TestIntegration_E2E_' \ - ./internal/openclaw/ - -# Phase 6: Tunnel + Ollama (slow, ~8min, needs Ollama model cached) -go test -tags integration -v -timeout 15m \ - -run 'TestIntegration_Tunnel_' \ - ./internal/openclaw/ - -# Phase 7: Fork validation (medium, ~5min) -go test -tags integration -v -timeout 10m \ - -run 'TestIntegration_Fork_FullPaymentFlow|TestIntegration_Fork_AgentSkillIteration' \ - ./internal/openclaw/ - -# Phase 5+: Real facilitator (medium, ~5min, needs x402-rs) -export X402_RS_DIR=/path/to/x402-rs -go test -tags integration -v -timeout 15m \ - -run 'TestIntegration_Fork_RealFacilitatorPayment' \ - ./internal/openclaw/ - -# Phase 8: FULL — tunnel + Ollama + real facilitator (~8min, needs everything) -export X402_RS_DIR=/path/to/x402-rs -go test -tags integration -v -timeout 15m \ - -run 'TestIntegration_Tunnel_RealFacilitatorOllama' \ - ./internal/openclaw/ - -# x402 verifier standalone E2E -go test -tags integration -v -timeout 10m \ - -run 'TestIntegration_PaymentGate' \ - ./internal/x402/ - -# All monetize tests -go test -tags integration -v -timeout 20m ./internal/openclaw/ -``` diff --git a/docs/monetisation-architecture-proposal.md b/docs/monetisation-architecture-proposal.md deleted file mode 100644 index ebcba0ff..00000000 --- a/docs/monetisation-architecture-proposal.md +++ /dev/null @@ -1,483 +0,0 @@ -# Obol Agent: Autonomous Compute Monetization - -**Branch:** `feat/secure-enclave-inference` | **Date:** 2026-02-25 | **Status:** Architecture proposal - -> Historical design note: the current implementation uses an event-driven `serviceoffer-controller`, `RegistrationRequest`, ServiceOffer-direct verifier watches, and controller finalizers. -> References below to the obol-agent-owned reconcile loop, OpenClaw cron jobs, or direct `x402-pricing` route mutation are superseded. - ---- - -## 1. The Goal - -A singleton OpenClaw instance — the **obol-agent** — deployed via `obol agent init`, autonomously monetizes compute resources running in the Obol Stack. A user (or the frontend) declares *what* to expose via a Custom Resource; the obol-agent handles *everything else*: model pulling, health validation, payment gating, public exposure, on-chain registration, and status reporting. - -No separate controller binary. No Go operator. The obol-agent is a regular OpenClaw instance with elevated RBAC and the `monetize` skill. Only one obol-agent can exist per cluster; other OpenClaw instances retain standard read-only access. - ---- - -## 2. How It Works - -``` - ┌──────────────────────────────────┐ - │ User / Frontend / obol CLI │ - │ │ - │ kubectl apply -f offer.yaml │ - │ OR: frontend POST to k8s API │ - │ OR: obol sell http ... │ - └──────────┬───────────────────────────┘ - │ creates CR - ▼ - ┌────────────────────────────────────┐ - │ ServiceOffer CR │ - │ apiVersion: obol.org/v1alpha1 │ - │ kind: ServiceOffer │ - └──────────┬───────────────────────────┘ - │ read by - ▼ - ┌────────────────────────────────────┐ - │ obol-agent (singleton OpenClaw) │ - │ namespace: openclaw- │ - │ │ - │ Cron job (every 60s): │ - │ python3 monetize.py process --all│ - │ │ - │ `monetize` skill: │ - │ 1. Read ServiceOffer CRs │ - │ 2. Pull model (if runtime=ollama) │ - │ 3. Health-check upstream service │ - │ 4. Create ForwardAuth Middleware │ - │ 5. Create HTTPRoute │ - │ 6. Register on ERC-8004 │ - │ 7. Update CR status │ - └────────────────────────────────────┘ -``` - -The obol-agent uses its mounted ServiceAccount token to talk to the Kubernetes API — the same pattern `kube.py` already uses for read-only monitoring, but extended with write operations for Middleware and HTTPRoute resources. - -The reconciliation loop is built on OpenClaw's native **cron system**: a `{ kind: "every", everyMs: 60000 }` job runs `monetize.py process --all` every 60 seconds. No sidecar, no K8s CronJob — the cron scheduler runs inside the OpenClaw Gateway process and persists across pod restarts. - ---- - -## 3. Why Not a Separate Controller - -| Concern | Go operator (controller-runtime) | OpenClaw with `monetize` skill | -|---------|----------------------------------|--------------------------------| -| New binary to build/maintain | Yes — new cmd/, Dockerfile, CI | No — skill is a SKILL.md + Python script | -| Hot-updatable logic | No — rebuild + redeploy image | Yes — update skill files on PVC | -| Error handling | Hardcoded retry/backoff | AI reasons about failures, adapts | -| Watch loop | Built-in informer cache | Built-in cron: `monetize.py process --all` every 60s | -| Dependencies | controller-runtime, kubebuilder, code-gen | stdlib Python (`urllib`, `json`, `ssl`) | -| Existing infrastructure | Needs new Deployment, SA, RBAC | Uses existing OpenClaw pod, SA, skill system | - -The traditional operator pattern is the right answer when you need guaranteed sub-second reconciliation with leader election. For monetization lifecycle (deploy → expose → register → monitor), OpenClaw acting on ServiceOffer CRs via skills is simpler and leverages everything already built. - ---- - -## 4. The CRD - -```yaml -apiVersion: obol.org/v1alpha1 -kind: ServiceOffer -metadata: - name: qwen-inference - namespace: openclaw-default # lives alongside the OpenClaw instance -spec: - # What to serve - model: - name: Qwen/Qwen3.5-35B-A3B # Ollama model tag to pull - runtime: ollama # runtime that serves the model - - # Upstream service (Ollama already running in-cluster) - upstream: - service: ollama # k8s Service name - namespace: openclaw-default # where the service runs - port: 11434 - healthPath: /api/tags # endpoint to probe after pull - - # How to price it - pricing: - amount: "0.50" - unit: MTok # per million tokens - currency: USDC - chain: base - - # Who gets paid - wallet: "0x1234...abcd" - - # Public path - path: /services/qwen-inference - - # On-chain advertisement - register: true -``` - -```yaml -status: - conditions: - - type: ModelReady - status: "True" - reason: PullCompleted - message: "Qwen/Qwen3.5-35B-A3B pulled and loaded on ollama" - - type: UpstreamHealthy - status: "True" - reason: HealthCheckPassed - message: "Model responds to inference at ollama.openclaw-default.svc:11434" - - type: PaymentGateReady - status: "True" - reason: MiddlewareCreated - message: "ForwardAuth middleware x402-qwen-inference created" - - type: RoutePublished - status: "True" - reason: HTTPRouteCreated - message: "Exposed at /services/qwen-inference via traefik-gateway" - - type: Registered - status: "True" - reason: ERC8004Registered - message: "Registered on Base (tx: 0xabc...)" - - type: Ready - status: "True" - reason: AllConditionsMet - endpoint: "https://stack.example.com/services/qwen-inference" - observedGeneration: 1 -``` - -**Design:** -- **Namespace-scoped** — the CR lives in the same namespace as the upstream service. This preserves OwnerReference cascade (garbage collection on delete) and avoids cross-namespace complexity. The obol-agent's ClusterRoleBinding lets it watch ServiceOffers across all namespaces via `GET /apis/obol.org/v1alpha1/serviceoffers` (cluster-wide list). -- **Conditions, not Phase** — [deprecated by API conventions](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#typical-status-properties). Conditions give granular insight into which step failed. -- **Status subresource** — prevents users from accidentally overwriting status. ([docs](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#status-subresource)) -- **Same-namespace as upstream** — the Middleware and HTTPRoute are created alongside the upstream service. OwnerReferences work (same namespace), so deleting the ServiceOffer garbage-collects the route and middleware. ([docs](https://kubernetes.io/docs/concepts/overview/working-with-objects/owners-dependents/)) - -### CRD installation - -The CRD manifest is embedded in the infrastructure helmfile (same pattern as `obol-agent.yaml`) and applied during `obol stack init`. No kubebuilder, no code-gen — just a static YAML manifest. - ---- - -## 5. The `monetize` Skill - -``` -internal/embed/skills/monetize/ -├── SKILL.md # Teaches OpenClaw when and how to use this skill -├── scripts/ -│ └── monetize.py # K8s API client for ServiceOffer lifecycle -└── references/ - └── x402-pricing.md # Pricing strategies, chain selection -``` - -### SKILL.md (summary) - -Teaches OpenClaw: -- When a user asks to monetize a service, create a ServiceOffer CR -- When asked to check monetization status, read ServiceOffer CRs and report conditions -- When asked to process offers, run the monetization workflow (health → gate → route → register) -- When asked to stop monetizing, delete the ServiceOffer CR (garbage collection handles cleanup) - -### kube.py extension - -`kube.py` gains write helpers (`api_post`, `api_patch`, `api_delete`) alongside its existing `api_get`. The read-only contract is preserved by convention: `kube.py` commands remain read-only; `monetize.py` imports the shared helpers and adds write operations. Pure Python stdlib — no new dependencies. - -Why not a K8s MCP server? The mounted ServiceAccount token already gives direct API access. An MCP server (e.g., Red Hat's `containers/kubernetes-mcp-server`) adds a sidecar container, image pull, and Helm chart changes for what amounts to wrapping the same REST calls. It's a known upgrade path if K8s operations outgrow script-based tooling, but adds no value today. - -### monetize.py - -``` -python3 monetize.py offers # list ServiceOffer CRs -python3 monetize.py process # run full workflow for one offer -python3 monetize.py process --all # process all pending offers -python3 monetize.py status # show conditions -python3 monetize.py create --upstream .. # create a ServiceOffer CR -python3 monetize.py delete # delete CR (cascades cleanup) -``` - -Each `process` invocation: - -1. **Read the ServiceOffer CR** from the k8s API -2. **Pull the model** — if `spec.model.runtime == ollama`, `POST /api/pull` to Ollama -3. **Health-check** — verify model responds at `..svc:` -4. **Create/update Middleware** — Traefik ForwardAuth pointing at `x402-verifier.x402.svc:8080/verify` -5. **Create/update HTTPRoute** — `parentRef: traefik-gateway`, path from spec, backend = upstream service, filter = the Middleware -6. **ERC-8004 registration** — if `spec.register`, call `signer.py` to sign and submit the registration tx -7. **Update CR status** — set conditions and endpoint - -All via the k8s REST API using the mounted ServiceAccount token. No kubectl, no client-go, no external dependencies. - ---- - -## 6. What Gets Created Per ServiceOffer - -All resources are created in the **same namespace** as the upstream service (and the ServiceOffer CR). OwnerReferences on the ServiceOffer handle cleanup. - -| Resource | Purpose | -|----------|---------| -| `Middleware` (traefik.io/v1alpha1) | ForwardAuth to `x402-verifier.x402.svc:8080/verify` — gates the upstream with payment | -| `HTTPRoute` (gateway.networking.k8s.io/v1) | Routes `spec.path` from Traefik Gateway to upstream, through the Middleware | - -That's it. Two resources. The upstream service already runs. The x402 verifier already runs. The Gateway already runs. The tunnel already runs. - -### Why no new namespace - -The upstream service already has a namespace. Creating a new namespace per offer would mean: -- Cross-namespace OwnerReferences don't work ([docs](https://kubernetes.io/docs/concepts/overview/working-with-objects/owners-dependents/)) -- Need ReferenceGrant for cross-namespace backend refs in HTTPRoute ([docs](https://gateway-api.sigs.k8s.io/api-types/referencegrant/)) -- Broader RBAC (namespace create/delete permissions) - -Instead: Middleware and HTTPRoute live alongside the upstream. Delete the ServiceOffer CR → Kubernetes cascades the deletion. - -### Cross-namespace HTTPRoute → Gateway - -The HTTPRoute references `traefik-gateway` in the `traefik` namespace. No ReferenceGrant needed — the Gateway's `allowedRoutes.namespaces.from: All` handles this. ([Gateway API docs](https://gateway-api.sigs.k8s.io/guides/multiple-ns/)) - -### Middleware locality - -Traefik's `ExtensionRef` in HTTPRoute is a `LocalObjectReference` — Middleware must be in the same namespace as the HTTPRoute. The skill creates it there. ([traefik#11126](https://github.com/traefik/traefik/issues/11126)) - ---- - -## 7. RBAC: Singleton obol-agent vs Regular OpenClaw - -### Two tiers of access - -| | obol-agent (singleton) | Regular OpenClaw instances | -|---|---|---| -| **Deployed by** | `obol agent init` | `obol openclaw onboard` | -| **RBAC** | `openclaw-monetize` ClusterRole | Namespace-scoped read-only Role (chart default) | -| **Skills** | All default skills + `monetize` | Default skills only | -| **Cron** | `monetize.py process --all` every 60s | No monetization cron | -| **Count** | Exactly one per cluster | Zero or more | - -Only the obol-agent gets the elevated ClusterRole. `obol agent init` enforces the singleton constraint — it refuses to create a second obol-agent if one already exists. - -### obol-agent ClusterRole - -```yaml -apiVersion: rbac.authorization.k8s.io/v1 -kind: ClusterRole -metadata: - name: openclaw-monetize -rules: - # Read/write ServiceOffer CRs - - apiGroups: ["obol.org"] - resources: ["serviceoffers"] - verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] - - apiGroups: ["obol.org"] - resources: ["serviceoffers/status"] - verbs: ["get", "update", "patch"] - - # Create Middleware and HTTPRoute in service namespaces - - apiGroups: ["traefik.io"] - resources: ["middlewares"] - verbs: ["get", "list", "create", "update", "patch", "delete"] - - apiGroups: ["gateway.networking.k8s.io"] - resources: ["httproutes"] - verbs: ["get", "list", "create", "update", "patch", "delete"] - - # Read pods/services/endpoints/deployments for health checks (any namespace) - - apiGroups: [""] - resources: ["pods", "services", "endpoints"] - verbs: ["get", "list"] - - apiGroups: ["apps"] - resources: ["deployments"] - verbs: ["get", "list"] - - apiGroups: [""] - resources: ["pods/log"] - verbs: ["get"] -``` - -This is bound to OpenClaw's ServiceAccount via ClusterRoleBinding — the skill needs to read services and create routes across namespaces (e.g., check health of Ollama in `openclaw-default`, create a route for an Ethereum node in `ethereum-knowing-wahoo`). - -### What is explicitly NOT granted - -| Excluded | Why | -|----------|-----| -| `secrets` (cluster-wide) | OpenClaw has secrets access in its own namespace only (chart default) | -| `rbac.authorization.k8s.io/*` | Cannot modify its own permissions | -| `namespaces` create/delete | Doesn't create namespaces | -| `deployments` create/update | Doesn't create workloads — gates existing ones | -| `configmaps` create (cluster-wide) | Reads config for diagnostics, doesn't write it | - -### How this gets applied - -The ClusterRole and ClusterRoleBinding are added to the OpenClaw helmfile generation in `internal/openclaw/openclaw.go`, same as the existing `rbac.create: true` overlay. When `obol openclaw onboard` runs, the chart deploys these RBAC resources alongside the pod. - -**Ref:** [RBAC Good Practices](https://kubernetes.io/docs/concepts/security/rbac-good-practices/) - -### Fix the existing `admin` RoleBinding - -The per-network `agent-rbac.yaml` currently binds the `admin` ClusterRole, which includes Secrets and RBAC manipulation. Replace with a scoped ClusterRole (read pods/services + write Middleware/HTTPRoute). - ---- - -## 8. Admission Policy Guardrail - -Defense-in-depth via [ValidatingAdmissionPolicy](https://kubernetes.io/docs/reference/access-authn-authz/validating-admission-policy/) (GA in k8s 1.30, available in k3s 1.31): - -```yaml -apiVersion: admissionregistration.k8s.io/v1 -kind: ValidatingAdmissionPolicy -metadata: - name: openclaw-monetize-guardrail -spec: - failurePolicy: Fail - matchConstraints: - resourceRules: - - apiGroups: ["traefik.io"] - apiVersions: ["v1alpha1"] - operations: ["CREATE", "UPDATE"] - resources: ["middlewares"] - - apiGroups: ["gateway.networking.k8s.io"] - apiVersions: ["v1"] - operations: ["CREATE", "UPDATE"] - resources: ["httproutes"] - matchConditions: - - name: is-openclaw - expression: >- - request.userInfo.username.startsWith("system:serviceaccount:openclaw-") - validations: - # HTTPRoutes must reference traefik-gateway only - - expression: >- - object.spec.parentRefs.all(ref, - ref.name == "traefik-gateway" && ref.?namespace.orValue("traefik") == "traefik" - ) - message: "OpenClaw can only attach routes to traefik-gateway" - # Middlewares must use ForwardAuth to x402-verifier only - - expression: >- - !has(object.spec.forwardAuth) || - object.spec.forwardAuth.address.startsWith("http://x402-verifier.x402.svc") - message: "ForwardAuth must point to x402-verifier" -``` - -Even if RBAC allows creating any Middleware, the admission policy ensures OpenClaw can only create ForwardAuth rules pointing at the legitimate x402 verifier. A prompt injection can't make it route traffic to an attacker-controlled auth endpoint. - ---- - -## 9. The Full Flow - -``` -1. User: "Monetize Qwen3.5-35B-A3B on Ollama at $0.50 per M token on Base" - -2. OpenClaw (using monetize skill) creates the ServiceOffer CR: - python3 monetize.py create qwen-inference \ - --model Qwen/Qwen3.5-35B-A3B --runtime ollama \ - --upstream ollama --namespace openclaw-default --port 11434 \ - --price 0.50 --unit MTok --chain base --wallet 0x... --register - → Creates ServiceOffer CR via k8s API - -3. OpenClaw processes the offer: - python3 monetize.py process qwen-inference - - Step 1: Pull the model through Ollama - POST http://ollama.openclaw-default.svc:11434/api/pull - {"name": "Qwen/Qwen3.5-35B-A3B"} - → Streams download progress, waits for completion - → sets condition: ModelReady=True - - Step 2: Health-check the model is loaded - POST http://ollama.openclaw-default.svc:11434/api/generate - {"model": "Qwen/Qwen3.5-35B-A3B", "prompt": "ping", "stream": false} - → 200 OK, model responds - → sets condition: UpstreamHealthy=True - - Step 3: Create ForwardAuth Middleware - POST /apis/traefik.io/v1alpha1/namespaces/openclaw-default/middlewares - → ForwardAuth → x402-verifier.x402.svc:8080/verify - → sets condition: PaymentGateReady=True - - Step 4: Create HTTPRoute - POST /apis/gateway.networking.k8s.io/v1/namespaces/openclaw-default/httproutes - → parentRef: traefik-gateway, path: /services/qwen-inference - → filter: ExtensionRef to Middleware - → backendRef: ollama:11434 - → sets condition: RoutePublished=True - - Step 5: ERC-8004 registration - python3 signer.py ... (signs registration tx) - → sets condition: Registered=True - - Step 6: Update status - PATCH /apis/obol.org/v1alpha1/.../serviceoffers/qwen-inference/status - → Ready=True, endpoint=https://stack.example.com/services/qwen-inference - -4. User: "What's the status?" - python3 monetize.py status qwen-inference - → Shows conditions table + endpoint + model info - -5. External consumer pays and calls: - POST https://stack.example.com/services/qwen-inference/v1/chat/completions - X-Payment: - → Traefik → ForwardAuth (x402-verifier) → Ollama (Qwen3.5-35B-A3B) -``` - ---- - -## 10. What the `obol` CLI Does - -The CLI becomes a thin CRD client — no deployment logic, no helmfile: - -```bash -obol sell http --upstream ollama --price 0.001 --chain base -# → creates ServiceOffer CR (same as kubectl apply) - -obol sell list -# → kubectl get serviceoffers (formatted) - -obol sell status qwen-inference -# → shows conditions, endpoint, pricing - -obol sell delete qwen-inference -# → deletes CR (OwnerReference cascades cleanup) -``` - -The frontend can do the same via the k8s API directly. - ---- - -## 11. What We Keep, What We Drop, What We Add - -| Component | Action | Reason | -|-----------|--------|--------| -| `cmd/x402-verifier/` | **Keep** | ForwardAuth verifier — the payment gate | -| `internal/x402/` | **Keep** | Verifier handler | -| `internal/erc8004/` | **Keep** | On-chain registration (called by `monetize.py` via `signer.py`) | -| `internal/enclave/` | **Keep** | Secure Enclave signing (orthogonal to monetization) | -| `internal/inference/gateway.go` | **Drop** | Inline x402 middleware — replaced by ForwardAuth | -| `internal/inference/store.go` | **Drop** | Deployment config on disk — replaced by CRD | -| `obol-agent.yaml` (busybox pod) | **Drop** | OpenClaw IS the agent; no separate placeholder pod | -| `agent-rbac.yaml` (`admin` binding) | **Replace** | Scoped ClusterRole instead of `admin` | -| `cmd/obol/service.go` | **Simplify** | Thin CRD client | -| `cmd/obol/monetize.go` | **Simplify** | Thin CRD client | -| `internal/embed/skills/monetize/` | **Add** | New skill: SKILL.md + `monetize.py` + references | -| ServiceOffer CRD manifest | **Add** | Intent interface, applied during `obol stack init` | -| ValidatingAdmissionPolicy | **Add** | Guardrail on what OpenClaw can create | -| `openclaw-monetize` ClusterRole | **Add** | Scoped write access for Middleware/HTTPRoute | - ---- - -## 12. Resolved Decisions - -| Question | Decision | Rationale | -|----------|----------|-----------| -| **Polling vs event-driven** | OpenClaw cron job, every 60s | OpenClaw has a built-in cron scheduler (`{ kind: "every", everyMs: 60000 }`). No sidecar, no K8s CronJob — runs inside the Gateway process. Jobs persist across restarts via `~/.openclaw/cron/jobs.json`. | -| **Multi-instance** | Singleton obol-agent | Only one obol-agent per cluster, enforced by `obol agent init`. Other OpenClaw instances keep read-only RBAC and no `monetize` skill. No coordination problem. | -| **CRD scope** | Namespace-scoped | OwnerReference cascade works (same namespace as Middleware/HTTPRoute). The obol-agent's ClusterRoleBinding lets it list ServiceOffers across all namespaces. Standard `kubectl get serviceoffers -A` works. | -| **K8s API access** | Extend `kube.py` with write helpers | `kube.py` gains `api_post`, `api_patch`, `api_delete` alongside `api_get`. `monetize.py` imports the shared helpers. Pure stdlib, zero new dependencies. K8s MCP server (Red Hat `containers/kubernetes-mcp-server`) is a known upgrade path but unnecessary today. | - ---- - -## References - -| Topic | Link | -|-------|------| -| Custom Resource Definitions | https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/ | -| CRD status subresource | https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#status-subresource | -| API conventions (conditions) | https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md | -| RBAC | https://kubernetes.io/docs/reference/access-authn-authz/rbac/ | -| RBAC good practices | https://kubernetes.io/docs/concepts/security/rbac-good-practices/ | -| ValidatingAdmissionPolicy | https://kubernetes.io/docs/reference/access-authn-authz/validating-admission-policy/ | -| OwnerReferences | https://kubernetes.io/docs/concepts/overview/working-with-objects/owners-dependents/ | -| Cross-namespace routing (Gateway API) | https://gateway-api.sigs.k8s.io/guides/multiple-ns/ | -| ReferenceGrant | https://gateway-api.sigs.k8s.io/api-types/referencegrant/ | -| Accessing API from a pod | https://kubernetes.io/docs/tasks/run-application/access-api-from-pod/ | -| Pod Security Standards | https://kubernetes.io/docs/concepts/security/pod-security-standards/ | -| Service account tokens | https://kubernetes.io/docs/concepts/security/service-accounts/ | -| Traefik ForwardAuth | https://doc.traefik.io/traefik/reference/routing-configuration/http/middlewares/forwardauth/ | -| Traefik Middleware locality | https://github.com/traefik/traefik/issues/11126 | diff --git a/docs/plans/buy-side-testing.md b/docs/plans/buy-side-testing.md deleted file mode 100644 index 39bfc260..00000000 --- a/docs/plans/buy-side-testing.md +++ /dev/null @@ -1,214 +0,0 @@ -# Buy-Side x402 Hands-Off Testing Plan - -## Current State - -- All clusters are down, no k3d containers running -- x402 extension (`x402.py`) created in LiteLLM fork, registered in `__init__.py` -- `buy-inference` skill created: `buy.py` + `SKILL.md` + `references/x402-buyer-api.md` -- `buy_side_test.go` exists but bypasses LiteLLM (sends directly to mock seller) -- LiteLLM Docker image `latest` includes x402 extension - -## Gaps (ordered by dependency) - -### Gap 0: LiteLLM image with x402 extension - -**Problem**: The LiteLLM Docker image needs to include the x402 extension for buy-side payments. - -**Fix**: -1. Ensure `internal/embed/infrastructure/base/templates/llm.yaml` references the correct LiteLLM image tag -2. The LiteLLM image should include x402 extension support -3. Update `llm.yaml` to use the correct version if needed - -**Verification**: `docker run --rm litellm python -c "from litellm.extensions.providers.x402 import install_x402; print('ok')"` (if applicable) - ---- - -### Gap 1: No test routes through LiteLLM x402 extension - -**Problem**: `buy_side_test.go` patches the ConfigMap but sends the paid request directly to the mock seller at `http://127.0.0.1:`. The critical path — LiteLLM receiving a request, the x402 extension signing via remote-signer, injecting `X-PAYMENT`, forwarding to the seller — is never exercised. - -**Fix**: Add a new integration test `TestIntegration_BuySide_ThroughLiteLLM` that: - -1. Starts mock x402 seller on host (reuse `startMockX402Seller`) -2. Patches `litellm-config` ConfigMap with x402 provider pointing at mock seller -3. Restarts litellm deployment to force immediate reload (not wait 120s) -4. Port-forwards litellm:4000 to localhost -5. Sends a chat request to litellm with the purchased model name (e.g., `test-buy-x402/test-model`) -6. litellm routes to `X402Provider.chat()` → signs via remote-signer → injects X-PAYMENT → forwards to mock seller -7. Asserts: mock seller received the X-PAYMENT header, response is 200 with inference data - -**Requires**: Running cluster with litellm + remote-signer (from `obol openclaw onboard`) - -**Key detail**: The mock seller must be reachable from inside the cluster. Use `testutil.ClusterHostIP(t)` (resolves to `host.k3d.internal` or `host.docker.internal`). Listen on `0.0.0.0` (already done in `startMockX402Seller`). - ---- - -### Gap 2: No mock remote-signer for isolated testing - -**Problem**: The x402 extension calls `POST remote-signer:9000/api/v1/sign/{addr}/typed-data`. In a full cluster, the real remote-signer handles this. But for faster/lighter tests, we have no mock. - -**Fix**: Add `testutil.StartMockRemoteSigner(t, privateKeyHex)` to provide a mock remote-signer that: - -1. Listens on `0.0.0.0:` -2. `GET /api/v1/keys` → returns `{"keys": ["
"]}` -3. `GET /healthz` → returns `{"status": "ok"}` -4. `POST /api/v1/sign/{addr}/typed-data` → uses `go-ethereum` crypto to sign EIP-712 typed data with the provided private key → returns `{"signature": "0x..."}` - -**Why**: Enables testing the LiteLLM x402 extension → remote-signer path without deploying the Rust remote-signer binary. Also enables testing `buy.py` commands (`balance` excepted) without a full cluster. - -**Scope**: ~80 lines Go. Reuses `testutil.eip712_signer.go` for signing logic. - -**Priority**: NICE-TO-HAVE for first test pass. The real remote-signer works fine in-cluster. Only needed if we want to test without a full cluster later. - ---- - -### Gap 3: buy.py skill not smoke-tested in-pod - -**Problem**: `buy.py` imports from sibling skills (`kube.py`, `signer.py`) via `sys.path.insert`. This works in theory (same pattern as `monetize.py`) but has never been tested in an actual pod where the skills are deployed at `/data/.openclaw/skills/`. - -**Fix**: Add a smoke test to verify the buy-inference skill loads correctly in-pod: - -```python -def test_buy_inference_help(): - """buy-inference skill loads and prints help.""" - result = subprocess.run( - ["python3", "/data/.openclaw/skills/buy-inference/scripts/buy.py", "--help"], - capture_output=True, text=True, timeout=10, - ) - assert result.returncode == 0 - assert "probe" in result.stdout - assert "buy" in result.stdout -``` - -**Scope**: 10 lines. - ---- - -### Gap 4: `llm.yaml` image tag configuration - -**Problem**: `internal/embed/infrastructure/base/templates/llm.yaml` needs to reference the correct LiteLLM image with x402 support. - -**Fix**: Ensure the LiteLLM deployment in `llm.yaml` uses the correct image tag: -```yaml -image: litellm:latest # or appropriate version with x402 support -``` - -**Scope**: Verify image references in llm.yaml are correct. - ---- - -## Testing Sequence - -### Phase 1: Build & Push (pre-cluster) - -``` -1. Ensure LiteLLM image with x402 extension is available (Gap 0) -2. Update llm.yaml image tag (Gap 4) -3. Build obol binary from worktree -4. Verify: go build ./... && go test ./... && go vet -tags integration ./internal/x402/ -``` - -### Phase 2: Cluster Up - -``` -5. OBOL_DEVELOPMENT=true obol stack init && obol stack up -6. obol openclaw onboard (deploys remote-signer + agent) -7. Verify: kubectl get pods -n llm (litellm Running) -8. Verify: kubectl get pods -n openclaw-obol-agent (remote-signer Running) -``` - -### Phase 3: Buy Skill Smoke Test - -``` -9. kubectl exec -n openclaw-obol-agent deploy/openclaw -- \ - python3 /data/.openclaw/skills/buy-inference/scripts/buy.py --help -10. kubectl exec -n openclaw-obol-agent deploy/openclaw -- \ - python3 /data/.openclaw/skills/buy-inference/scripts/buy.py list - (expect: "No purchased x402 providers.") -``` - -### Phase 4: Manual Buy-Side Walkthrough - -``` -11. Start mock seller on host: - go test -tags integration -v -run TestIntegration_BuySide_ProbeAndPurchase -timeout 10m ./internal/x402/ - (or start a real seller via: obol sell inference on another cluster) - -12. From inside the agent pod, run probe: - kubectl exec -n openclaw-obol-agent deploy/openclaw -- \ - python3 /data/.openclaw/skills/buy-inference/scripts/buy.py probe \ - http://host.k3d.internal:/v1/chat/completions - (expect: 402 pricing output) - -13. From inside the agent pod, run buy: - kubectl exec -n openclaw-obol-agent deploy/openclaw -- \ - python3 /data/.openclaw/skills/buy-inference/scripts/buy.py buy test-seller \ - --endpoint http://host.k3d.internal: \ - --model test-model --budget 10000 - (expect: provider added to litellm-config) - -14. Wait 2 min for ConfigMap reload, or force: - kubectl rollout restart -n llm deploy/litellm - kubectl rollout status -n llm deploy/litellm --timeout=60s - -15. Verify model appears in litellm: - kubectl exec -n llm deploy/litellm -- curl -s http://localhost:4000/models | jq . - -16. Send inference through litellm using purchased model: - kubectl exec -n llm deploy/litellm -- curl -s -X POST http://localhost:4000/v1/chat/completions \ - -H "Content-Type: application/json" \ - -d '{"model":"test-seller/test-model","messages":[{"role":"user","content":"hello"}]}' - (expect: x402 extension signs payment, forwards to seller, returns 200) - -17. Check seller received X-PAYMENT header (from test logs or mock seller output) - -18. Cleanup: - kubectl exec -n openclaw-obol-agent deploy/openclaw -- \ - python3 /data/.openclaw/skills/buy-inference/scripts/buy.py remove test-seller -``` - -### Phase 5: Integration Test (automated) - -``` -19. Run the through-litellm integration test (Gap 1): - go test -tags integration -v -run TestIntegration_BuySide_ThroughLiteLLM -timeout 10m ./internal/x402/ - -20. Run existing buy-side tests: - go test -tags integration -v -run TestIntegration_BuySide -timeout 10m ./internal/x402/ -``` - -### Phase 6: Full Hands-Off (OpenClaw agent does it autonomously) - -``` -21. Trigger OpenClaw heartbeat with a task that exercises the buy skill: - "Discover x402 inference sellers, probe the first one, buy access if the price - is under 10000 micro-units, then send a test message through the purchased model." - -22. Watch logs for ~5 min: - kubectl logs -n openclaw-obol-agent deploy/openclaw -f - -23. Verify: the agent probed, bought, and used a remote model autonomously -``` - -## Minimal Critical Path - -If time is limited, the absolute minimum to verify the buy lifecycle works: - -1. **Gap 0** — ensure LiteLLM image with x402 extension is available (BLOCKER) -2. **Gap 4** — update image tag in llm.yaml (BLOCKER) -3. Build obol binary, bring up cluster, onboard openclaw -4. Start mock seller on host -5. Run `buy.py probe` + `buy.py buy` from agent pod -6. Restart litellm, send request through purchased model -7. Verify 200 response with X-PAYMENT header at seller - -Everything else (Gap 1 automated test, Gap 2 mock signer, Gap 3 smoke test) can follow after the manual walkthrough confirms the flow works. - -## Files to Modify - -| File | Change | Gap | -|------|--------|-----| -| `internal/embed/infrastructure/base/templates/llm.yaml` | Verify LiteLLM image tag | 4 | -| `internal/x402/buy_side_test.go` | Add `TestIntegration_BuySide_ThroughLiteLLM` | 1 | -| `internal/testutil/mock_signer.go` | New: mock remote-signer | 2 | -| `tests/skills_smoke_test.py` | Add buy-inference smoke test | 3 | diff --git a/docs/plans/cli-agent-readiness.md b/docs/plans/cli-agent-readiness.md deleted file mode 100644 index 90a5d7aa..00000000 --- a/docs/plans/cli-agent-readiness.md +++ /dev/null @@ -1,307 +0,0 @@ -# CLI Agent-Readiness Optimizations - -## Status - -**Implemented (this branch)**: -- Phase 1: Global `--output json` / `-o json` / `OBOL_OUTPUT=json` flag -- Phase 1: `OutputMode` + `IsJSON()` + `JSON()` on `internal/ui/UI` -- Phase 1: 11 commands refactored with typed JSON results (sell list/status/info, network list, model status/list, version, update, openclaw list, tunnel status) -- Phase 1: Human output redirected to stderr in JSON mode (stdout is clean JSON) -- Phase 2: `internal/validate/` package (Name, Namespace, WalletAddress, ChainName, Price, URL, Path, NoControlChars) -- Phase 2: Headless prompt paths — `Confirm`, `Select`, `Input`, `SecretInput` auto-resolve defaults in non-TTY/JSON mode -- Phase 2: `sell delete` migrated from raw `fmt.Scanln` to `u.Confirm()` -- Phase 6: `CONTEXT.md` — agent-facing context document - -- Phase 1D: `--from-json` on sell http, sell pricing, network add (`cmd/obol/input.go` helper) -- Phase 2B: `validate.Name()` wired into sell inference/http/stop/delete, `validate.URL()` in network add -- Phase 2C: model.go `promptModelPull()` migrated from bufio to `u.Select()`/`u.Input()`, openclaw onboard headless via `u.IsTTY() && !u.IsJSON()` - -**Deferred to follow-up**: -- Phase 3: `obol describe` schema introspection -- Phase 4: `--fields` field filtering -- Phase 5: `--dry-run` for mutating commands -- Phase 7: MCP surface (`obol mcp`) - -## Context - -The obol CLI is increasingly consumed by AI agents — Claude Code during development, OpenClaw agents in-cluster, and soon MCP clients. Today the CLI is human-optimized: colored output, spinners, interactive prompts, and hand-formatted tables. Agents need structured output, non-interactive paths, input hardening, and runtime introspection. This plan makes the CLI agent-ready while preserving human DX. - -**Strengths**: `internal/ui/` abstraction with TTY detection, `OutputMode` (human/json), `--verbose`/`--quiet`/`--output` global flags, `internal/schemas/` with JSON-tagged Go types, `internal/validate/` for input validation, `--force` pattern for non-interactive destructive ops, 23 SKILL.md files shipped in `internal/embed/skills/`, `CONTEXT.md` for agent consumption. - -**Remaining gaps**: `--from-json` for structured input, some `fmt.Printf` calls still bypass UI layer, `model.go` interactive prompts not fully migrated, `openclaw onboard` still hardwired `Interactive: true`, no schema introspection, no `--dry-run`, no field filtering, no MCP surface. - ---- - -## Phase 1: Global `--output json` + Raw JSON Input - -Structured output is table stakes. Raw JSON input (`--from-json`) is first-class — agents shouldn't have to translate nested structures into 15+ flags. - -### 1A. Extend UI struct with output mode - -**`internal/ui/ui.go`** — Add `OutputMode` type (`human`|`json`) and field to `UI` struct. Add `NewWithAllOptions(verbose, quiet bool, output OutputMode)`. Add `IsJSON() bool`. - -**`internal/ui/output.go`** — Add `JSON(v any) error` method that writes to stdout via `json.NewEncoder`. When `IsJSON()` is true, redirect `Info`/`Success`/`Detail`/`Print`/`Printf`/`Dim`/`Bold`/`Blank` to stderr (so agents get clean JSON on stdout, diagnostics on stderr). Suppress spinners in JSON mode. - -### 1B. Add global `--output` flag - -**`cmd/obol/main.go`** (lines 110-127) — Add `--output` / `-o` flag (`human`|`json`, env `OBOL_OUTPUT`, default `human`). Wire in `Before` hook to pass to `ui.NewWithAllOptions`. - -### 1C. Refactor commands to return typed results - -Don't just bolt JSON onto existing `fmt.Printf` calls. Refactor high-value commands to return typed data first, then format for human or JSON. This pays off twice: clean JSON output now, and reusable typed results for MCP later. - -**Audit note**: Raw `fmt.Printf` output is spread across `main.go:460` (version), `model.go:286` (tables), `network.go:188` (tables), and throughout `sell.go`. Each needs a return-data-then-format refactor. - -| Command | Strategy | Effort | -|---------|----------|--------| -| `sell list` | Switch kubectl arg from `-o wide` to `-o json` | Trivial | -| `sell status ` | Switch kubectl arg from `-o yaml` to `-o json` | Trivial | -| `sell status` (global) | Marshal `PricingConfig` + `store.List()` — currently raw `fmt.Printf` at `sell.go:463-498` | Medium | -| `sell info` | Already has `--json` (`sell.go:841`) — wire to global flag, deprecate local | Trivial | -| `network list` | `ListRPCNetworks()` returns `[]RPCNetworkInfo` — marshal it, but local node output also uses `fmt.Printf` at `network.go:188` | Medium | -| `model status` | Return provider status map as JSON — currently `fmt.Printf` tables at `model.go:286` | Medium | -| `model list` | `ListOllamaModels()` returns structured data | Low | -| `version` | `BuildInfo()` returns a string today — refactor to struct with fields (version, commit, date, go version) | Medium | -| `update` | Already has `--json` (`update.go:20`); wire to global flag, deprecate local | Trivial | -| `openclaw list` | Refactor to return data before formatting | Medium | -| `tunnel status` | Refactor to return data before formatting | Medium | - -### 1D. Raw JSON input (`--from-json`) - -Add `--from-json` flag to all commands that create resources. Accepts file path or `-` for stdin. Unmarshals into existing `internal/schemas/` types, validates, creates manifest. This is first-class, not an afterthought. - -| Command | Schema Type | Flags Bypassed | -|---------|-------------|----------------| -| `sell http` | `schemas.ServiceOfferSpec` | 15+ flags (wallet, chain, price, upstream, port, namespace, health-path, etc.) | -| `sell inference` | `schemas.ServiceOfferSpec` | 10+ flags | -| `sell pricing` | `schemas.PaymentTerms` | wallet, chain, facilitator | -| `network add` | New `RPCConfig` type | endpoint, chain-id, allow-writes | - -### Testing -- `internal/ui/ui_test.go`: OutputMode switching, JSON writes valid JSON to stdout, human methods go to stderr in JSON mode -- `cmd/obol/output_test.go`: `--output json` on each migrated command produces parseable JSON -- `cmd/obol/json_input_test.go`: `--from-json` with valid/invalid specs - ---- - -## Phase 2: Input Validation + Headless Paths - -Agents hallucinate inputs and can't answer interactive prompts. Fix both together. - -### 2A. New validation package - -**`internal/validate/validate.go`** (new) - -``` -Name(s) — k8s-safe: [a-z0-9][a-z0-9.-]*, no path traversal -Namespace(s) — same rules as Name -WalletAddress(s) — reuse x402verifier.ValidateWallet() pattern -ChainName(s) — from known set (base, base-sepolia, etc.) -Path(s) — no .., no %2e%2e, no control chars -Price(s) — valid decimal, positive -URL(s) — parseable, no control chars -NoControlChars(s) — reject \x00-\x1f except \n\t -``` - -### 2B. Wire into commands - -Add validation at the top of every action handler for positional args and key flags: -- **`cmd/obol/sell.go`**: name, wallet, chain, path, price, namespace, upstream URL -- **`cmd/obol/network.go`**: network name, custom RPC URL, chain ID -- **`cmd/obol/model.go`**: provider name, endpoint URL -- **`cmd/obol/openclaw.go`**: instance ID - -### 2C. Headless paths for interactive flows - -**`internal/ui/prompt.go`** — When `IsJSON() || !IsTTY()`: -- `Confirm` → return default value (no stdin read) -- `Select` → return error: "interactive selection unavailable; use --provider flag" -- `Input` → return default or error if no default -- `SecretInput` → return error: "use --api-key flag" - -**`cmd/obol/openclaw.go`** (line 36) — `openclaw onboard` is hardwired `Interactive: true`. Add a non-interactive path when all required flags are provided (`--id`, plus any other required inputs). Only fall through to interactive mode when flags are missing AND stdin is a TTY. - -**`cmd/obol/model.go`** (lines 62-84) — `model setup` enters interactive selection when `--provider` is omitted. In non-TTY/JSON mode, error with required flags instead. - -**`cmd/obol/model.go`** (lines 387-419) — `model pull` uses `bufio.NewReader(os.Stdin)` for interactive model selection. Same treatment. - -**`cmd/obol/sell.go`** (line 576-588) — `sell delete` confirmation uses raw `fmt.Scanln`. Migrate to `u.Confirm()` so the headless path is automatic. - -### Testing -- `internal/validate/validate_test.go`: Table-driven tests for path traversal variants, control char injection, valid inputs -- Test that `--output json` + missing required flags → clear error (not a hung prompt) -- Test that `openclaw onboard --id test -o json` works without interactive mode - ---- - -## Phase 3: Schema Introspection (`obol describe`) - -Let agents discover what the CLI accepts at runtime without parsing `--help` text. - -### 3A. Add `obol describe` command - -**`cmd/obol/describe.go`** (new) - -``` -obol describe # list all commands + flags as JSON -obol describe sell http # flags + ServiceOffer schema for that command -obol describe --schemas # dump resource schemas only -``` - -Walk urfave/cli's `*cli.Command` tree. For each command, emit: name, usage, flags (name, type, required, default, env vars, aliases), ArgsUsage. Output always JSON. - -### 3B. Schema registry - -**`internal/schemas/registry.go`** (new) — Map of schema names to JSON Schema generated from Go struct tags via `reflect`. Schemas: `ServiceOfferSpec`, `PaymentTerms`, `PriceTable`, `RegistrationSpec`. - -### 3C. Command metadata annotations - -Add `Metadata: map[string]any{"schema": "ServiceOfferSpec", "mutating": true}` to commands that create resources (sell http, sell inference, sell pricing). `obol describe` reads this and includes the schema in output. - -### Testing -- `cmd/obol/describe_test.go`: Valid JSON output, every command appears, schemas resolve, flag metadata matches actual flags - ---- - -## Phase 4: `--fields` Support - -Let agents limit response size to protect their context window. - -### 4A. Field mask implementation - -**`internal/ui/fields.go`** (new) — `FilterFields(data any, fields []string) any` using reflect on JSON tags. - -### 4B. Global `--fields` flag - -**`cmd/obol/main.go`** — Global `--fields` flag (comma-separated, requires `--output json`). Applied in `u.JSON()` before encoding. - -### Testing -- `--fields name,status` on `sell list -o json` returns only those fields -- `--fields` without `--output json` returns error - ---- - -## Phase 5: `--dry-run` for Mutating Commands - -Let agents validate before mutating. Safety rail. - -### 5A. Global `--dry-run` flag - -**`cmd/obol/main.go`** — Add `--dry-run` bool flag. - -### 5B. Priority commands - -| Command | Implementation | -|---------|---------------| -| `sell http` | Already builds manifest before `kubectlApply()` — return manifest instead of applying | -| `sell pricing` | Validate wallet/chain, show what would be written to ConfigMap | -| `network add` | Validate chain, show which RPCs would be added to eRPC config | -| `sell delete` | Validate name exists, show what would be deleted | - -Pattern: after validation, before execution, check `cmd.Root().Bool("dry-run")` and return a `DryRunResult{Command, Valid, WouldCreate, Manifest}` as JSON. - -### Testing -- `cmd/obol/dryrun_test.go`: `--dry-run sell http` returns manifest without kubectl apply, validation still runs in dry-run - ---- - -## Phase 6: Agent Context & Skills - -The 23 SKILL.md files are a strength, but there's no top-level `CONTEXT.md` encoding invariants agents can't intuit from `--help`. - -### 6A. Ship `CONTEXT.md` - -**`CONTEXT.md`** (repo root, also embedded in binary) — Agent-facing context file encoding: -- Always use `--output json` when parsing output programmatically -- Always use `--force` for non-interactive destructive operations -- Always use `--fields` on list commands to limit context window usage -- Always use `--dry-run` before mutating operations -- Use `obol describe ` to introspect flags and schemas -- Cluster commands require `OBOL_CONFIG_DIR` or a running stack (`obol stack up`) -- Payment wallet addresses must be 0x-prefixed, 42 chars -- Chain names: `base`, `base-sepolia` (not CAIP-2 format) - -### 6B. Update existing skills - -Review and update the 23 SKILL.md files to reference the new agent-friendly flags where relevant (e.g., the `sell` skill should mention `--from-json` and `--dry-run`). - ---- - -## Phase 7: MCP Surface (`obol mcp`) - -Expose the CLI as typed JSON-RPC tools over stdio. Depends on all previous phases. - -### 7A. New package `internal/mcp/` - -- `server.go` — MCP server over stdio using `github.com/mark3labs/mcp-go` -- `tools.go` — Tool definitions from the typed result functions built in Phase 1C (not by shelling out with `--output json`) -- `handlers.go` — Tool handlers that call the refactored return-typed-data functions directly - -### 7B. `obol mcp` command - -**`cmd/obol/mcp.go`** (new) — Starts MCP server. Exposes high-value tools only: -- sell: `sell_http`, `sell_list`, `sell_status`, `sell_pricing`, `sell_delete` -- network: `network_list`, `network_add`, `network_remove`, `network_status` -- model: `model_status`, `model_list`, `model_setup` -- openclaw: `openclaw_list`, `openclaw_onboard` -- utility: `version`, `update`, `tunnel_status` - -Excludes: kubectl/helm/k9s passthroughs, interactive-only commands, dangerous ops (stack purge/down). - -### Testing -- `internal/mcp/mcp_test.go`: Tool registration produces valid MCP definitions, stdin/stdout JSON-RPC round-trip - ---- - -## Key Files Summary - -| File | Changes | -|------|---------| -| `internal/ui/ui.go` | Add OutputMode, IsJSON(), NewWithAllOptions() | -| `internal/ui/output.go` | Add JSON() method, stderr redirect in JSON mode | -| `internal/ui/prompt.go` | Non-interactive behavior when JSON/non-TTY | -| `internal/ui/fields.go` | New — field mask filtering | -| `cmd/obol/main.go` | `--output`, `--dry-run`, `--fields` global flags + Before hook | -| `cmd/obol/sell.go` | JSON output, typed results, input validation, dry-run, --from-json, migrate Scanln to u.Confirm | -| `cmd/obol/network.go` | JSON output, typed results, input validation | -| `cmd/obol/model.go` | JSON output, typed results, input validation, headless paths | -| `cmd/obol/openclaw.go` | JSON output, typed results, input validation, headless onboard path | -| `cmd/obol/update.go` | Wire to global --output flag, deprecate local --json | -| `cmd/obol/describe.go` | New — schema introspection command | -| `cmd/obol/mcp.go` | New — `obol mcp` command | -| `internal/validate/validate.go` | New — input validation functions | -| `internal/schemas/registry.go` | New — JSON Schema generation from Go types | -| `internal/mcp/` | New package — MCP server, tools, handlers | -| `CONTEXT.md` | New — agent-facing context file | - -## Verification - -```bash -# Phase 1: JSON output + JSON input -obol sell list -o json | jq . -obol sell status -o json | jq . -obol version -o json | jq . -obol network list -o json | jq . -echo '{"upstream":{"service":"ollama","namespace":"llm","port":11434},...}' | obol sell http test --from-json - - -# Phase 2: Input validation + headless -obol sell http '../etc/passwd' --wallet 0x... --chain base-sepolia # should error -obol sell http 'valid-name' --wallet 'not-a-wallet' --chain base-sepolia # should error -echo '' | obol model setup -o json # should error with "use --provider flag", not hang - -# Phase 3: Schema introspection -obol describe | jq '.commands | length' -obol describe sell http | jq '.schema' - -# Phase 4: Fields -obol sell list -o json --fields name,namespace,status | jq . - -# Phase 5: Dry-run -obol sell http test-svc --wallet 0x... --chain base-sepolia --dry-run -o json | jq . - -# Phase 7: MCP -echo '{"jsonrpc":"2.0","method":"tools/list","id":1}' | obol mcp - -# Unit tests -go test ./internal/ui/ ./internal/validate/ ./internal/schemas/ ./internal/mcp/ ./cmd/obol/ -``` diff --git a/docs/plans/multi-network-sell.md b/docs/plans/multi-network-sell.md deleted file mode 100644 index 77b6bf01..00000000 --- a/docs/plans/multi-network-sell.md +++ /dev/null @@ -1,387 +0,0 @@ -# Multi-Network Sell Command + UX Improvements - -## Context - -The `obol sell` command currently only supports ERC-8004 registration on Base Sepolia, requires manual private key management via `--private-key-file`, and forces users to specify all flags explicitly. We want to: - -1. Support 3 registration networks: **base-sepolia**, **base**, **ethereum mainnet** -2. Support **multi-chain** registration: `--chain mainnet,base` registers on both, best-effort -3. Use the **remote-signer** for all signing (not private key extraction) — EIP-712 typed data + transaction signing via its REST API -4. Use **sponsored registration** (zero gas) on ethereum mainnet via howto8004.com -5. Use the **local eRPC** (`localhost/rpc`) for chain access instead of public RPCs -6. Add **interactive prompts** using `charmbracelet/huh` with good defaults -7. **Auto-discover** the remote-signer wallet address -8. Add **ethereum mainnet** as a valid x402 payment chain - -Frontend deferred to follow-up PR. EIP-7702 handled server-side by sponsor — no CLI implementation needed. - -### Network Matrix - -| Network | x402 Payment | x402 Facilitator | ERC-8004 Registration | Sponsored Reg | -|---------|-------------|-------------------|----------------------|---------------| -| base-sepolia | Yes | `facilitator.x402.rs` | Yes (direct tx via remote-signer) | No | -| base | Yes | `x402.gcp.obol.tech` | Yes (direct tx via remote-signer) | No | -| ethereum | Yes (no facilitator yet) | TBD | Yes | Yes (`sponsored.howto8004.com/api/register`) | - ---- - -## Phase 1: Multi-Network ERC-8004 Registry Config - -### `internal/erc8004/networks.go` (new) - -```go -type NetworkConfig struct { - Name string // "base-sepolia", "base", "ethereum" - ChainID int64 - RegistryAddress string // per-chain registry address - SponsorURL string // empty if no sponsor - DelegateAddress string // EIP-7702 delegate (for sponsored flow) - ERPCNetwork string // eRPC path segment: "base-sepolia", "base", "mainnet" -} - -func ResolveNetwork(name string) (NetworkConfig, error) -func ResolveNetworks(csv string) ([]NetworkConfig, error) // "mainnet,base" → []NetworkConfig -func SupportedNetworks() []NetworkConfig -``` - -Three entries: -- `base-sepolia`: chainID 84532, registry `0x8004A818BFB912233c491871b3d84c89A494BD9e`, eRPC `base-sepolia` -- `base`: chainID 8453, registry TBD (confirm CREATE2 address), eRPC `base` -- `ethereum` / `mainnet`: chainID 1, registry `0x8004A169FB4a3325136EB29fA0ceB6D2e539a432`, sponsor `https://sponsored.howto8004.com/api/register`, delegate `0x77fb3D2ff6dB9dcbF1b7E0693b3c746B30499eE8`, eRPC `mainnet` - -RPC URL is **not** in NetworkConfig — always use local eRPC at `http://localhost/rpc/{ERPCNetwork}` (from host via k3d port mapping). - -### `internal/erc8004/client.go` - -- Add `NewClientForNetwork(ctx, rpcBaseURL string, net NetworkConfig) (*Client, error)` — constructs RPC URL as `rpcBaseURL + "/" + net.ERPCNetwork`, uses `net.RegistryAddress` -- Keep `NewClient(ctx, rpcURL)` as backward-compat wrapper - -### Files -- `internal/erc8004/networks.go` (new) -- `internal/erc8004/networks_test.go` (new) -- `internal/erc8004/client.go` (add `NewClientForNetwork`) - ---- - -## Phase 2: Remote-Signer Integration for Registration - -### Architecture - -The remote-signer REST API at port 9000 already supports: -- `POST /api/v1/sign/{address}/transaction` — sign raw transactions -- `POST /api/v1/sign/{address}/typed-data` — sign EIP-712 typed data -- `GET /api/v1/keys` — list loaded wallet addresses - -From the host CLI, access via **temporary port-forward** to `remote-signer:9000` (same pattern as `openclaw cli`). - -### `internal/erc8004/signer.go` (new) - -```go -// RemoteSigner wraps the remote-signer REST API for ERC-8004 operations. -type RemoteSigner struct { - baseURL string // e.g. "http://localhost:19000" (port-forwarded) -} - -func NewRemoteSigner(baseURL string) *RemoteSigner - -// GetAddress returns the first loaded signing address. -func (s *RemoteSigner) GetAddress(ctx context.Context) (common.Address, error) - -// SignTransaction signs an EIP-1559 transaction for direct on-chain registration. -func (s *RemoteSigner) SignTransaction(ctx context.Context, addr common.Address, tx SignTxRequest) ([]byte, error) - -// SignTypedData signs EIP-712 typed data (for sponsored registration). -func (s *RemoteSigner) SignTypedData(ctx context.Context, addr common.Address, data EIP712TypedData) ([]byte, error) -``` - -### `internal/erc8004/register.go` (new) - -Two registration paths: - -**Direct on-chain** (base-sepolia, base): -1. Port-forward to remote-signer -2. `signer.GetAddress()` → wallet address -3. Build `register(agentURI)` calldata -4. Get nonce + gas estimates from eRPC -5. `signer.SignTransaction()` → signed tx -6. `eth_sendRawTransaction` via eRPC -7. Wait for receipt, parse `Registered` event - -**Sponsored** (ethereum mainnet): -1. Port-forward to remote-signer -2. `signer.GetAddress()` → wallet address -3. `signer.SignTypedData()` → EIP-712 authorization + registration intent signatures -4. POST to `net.SponsorURL` with signatures -5. Parse response `{success, agentId, txHash}` - -### Port-Forward Helper - -Reuse or adapt the pattern from `openclaw cli` (`cmd/obol/openclaw.go`). New helper: - -```go -// portForwardRemoteSigner starts a port-forward to the remote-signer in the -// given namespace and returns the local URL + cleanup function. -func portForwardRemoteSigner(cfg *config.Config, namespace string) (baseURL string, cleanup func(), err error) -``` - -### Files -- `internal/erc8004/signer.go` (new — remote-signer REST client) -- `internal/erc8004/signer_test.go` (new — HTTP mock tests) -- `internal/erc8004/register.go` (new — direct + sponsored registration flows) -- `internal/erc8004/sponsor.go` (new — sponsored API client, EIP-712 types) -- `internal/erc8004/sponsor_test.go` (new) - ---- - -## Phase 3: Wallet Auto-Discovery - -### `internal/openclaw/wallet_resolve.go` (new) - -```go -// ResolveWalletAddress returns the wallet address from the single OpenClaw instance. -// 0 instances → error, 1 → auto-select, 2+ → error suggesting --wallet. -func ResolveWalletAddress(cfg *config.Config) (string, error) - -// ResolveInstanceNamespace returns the namespace of the single OpenClaw instance -// (needed for port-forwarding to the remote-signer in that namespace). -func ResolveInstanceNamespace(cfg *config.Config) (string, error) -``` - -Flow: -1. `ListInstanceIDs(cfg)` → instance IDs -2. 0 → error, 1 → read wallet.json, 2+ → error with list of addresses -3. `ReadWalletMetadata(DeploymentPath(cfg, id))` → `WalletInfo.Address` - -**No private key extraction.** The address is all we need for auto-discovery. Signing goes through the remote-signer API. - -### Files -- `internal/openclaw/wallet_resolve.go` (new) -- `internal/openclaw/wallet_resolve_test.go` (new) - ---- - -## Phase 4: Rewrite `sell register` - -### `cmd/obol/sell.go` — `sellRegisterCommand` - -**New flags:** - -| Flag | Type | Default | Notes | -|------|------|---------|-------| -| `--chain` | string | `base-sepolia` | Comma-separated: `base-sepolia,base,mainnet`. Register on each, best-effort | -| `--sponsored` | bool | auto | `true` when network has sponsor URL | -| `--endpoint` | string | auto | Auto-detected from tunnel | -| `--name` | string | `Obol Agent` | Agent name for registration | -| `--description` | string | smart default | Auto-generated from stack info | -| `--image` | string | smart default | Default Obol logo URL | -| `--private-key-file` | string | | Fallback — used only if no remote-signer detected | - -**Removed:** `--private-key` (deprecated), `--rpc-url` (use local eRPC) - -**Action logic:** -1. Parse `--chain` → `erc8004.ResolveNetworks(chainCSV)` → `[]NetworkConfig` -2. Resolve wallet: try `openclaw.ResolveWalletAddress(cfg)`. If found, use remote-signer path. If not, require `--private-key-file`. -3. Resolve endpoint: `--endpoint` if set, else tunnel auto-detect -4. For each network (best-effort): - a. If sponsored + network has sponsor → sponsored path (sign EIP-712 via remote-signer, POST to sponsor) - b. Else → direct path (sign tx via remote-signer, broadcast via eRPC) - c. On success: print CAIP-10 registry line - d. On failure: print warning, continue to next chain -5. Update `agent-registration.json` with all successful registrations in the `registrations[]` array - -### Files -- `cmd/obol/sell.go` (rewrite `sellRegisterCommand`) -- `cmd/obol/sell_test.go` (update `TestSellRegister_Flags`) - ---- - -## Phase 5: Interactive Prompts with `charmbracelet/huh` - -### New dependency - -`go get github.com/charmbracelet/huh` - -### Signature change - -`sellCommand(cfg *config.Config)` → `sellCommand(cfg *config.Config, u *ui.UI)` (match `openclawCommand` pattern). Wire from `main.go`. - -### TTY guard - -```go -import "golang.org/x/term" -isInteractive := term.IsTerminal(int(os.Stdin.Fd())) -``` - -### `sell inference` interactive flow: - -| Field | Default | Prompt type | When prompted | -|-------|---------|-------------|---------------| -| Name | (required) | Text input | No positional arg | -| Model | (required) | Select from Ollama models | `--model` not set | -| Wallet | auto-discovered | Text (pre-filled) | Auto-discover fails | -| Chain | `base-sepolia` | Select | Using default | -| Price | `0.001` | Text (pre-filled) | Confirm or override | - -### `sell http` interactive flow: - -| Field | Default | Prompt type | When prompted | -|-------|---------|-------------|---------------| -| Name | (required) | Text input | No positional arg | -| Upstream | (required) | Text input | `--upstream` not set | -| Port | `8080` | Text (pre-filled) | Confirm | -| Wallet | auto-discovered | Text (pre-filled) | Auto-discover fails | -| Chain | `base-sepolia` | Select | `--chain` not set (remove `Required: true`) | -| Price model | `perRequest` | Select | No price flag set | -| Price value | `0.001` | Text | After model selected | -| Register? | `false` | Confirm | Not explicitly set | - -### `sell register` interactive flow: - -| Field | Default | Prompt type | When prompted | -|-------|---------|-------------|---------------| -| Chain(s) | `base-sepolia` | Multi-select | Using default | -| Name | `Obol Agent` | Text (pre-filled) | Confirm or override | -| Description | auto-generated | Text (pre-filled) | Confirm or override | -| Image | default logo URL | Text (pre-filled) | Confirm or override | -| Sponsored? | yes (when available) | Confirm | Network supports it | -| Endpoint | auto-detected | Text (pre-filled) | Tunnel fails | - -### Non-interactive path - -All prompts gated on `isInteractive`. When not TTY: flag validation applies, defaults used, no prompts. - -### Files -- `go.mod` / `go.sum` (add `charmbracelet/huh`) -- `cmd/obol/sell.go` (add prompts to inference, http, register) -- `cmd/obol/main.go` (wire `*ui.UI` to `sellCommand`) - ---- - -## Phase 6: x402 Payment Chain Updates - -### `cmd/obol/sell.go` — `resolveX402Chain` - -Add: -```go -case "ethereum", "ethereum-mainnet", "mainnet": - return x402.EthereumMainnet, nil -``` - -If `x402.EthereumMainnet` doesn't exist in the upstream `mark3labs/x402-go` library, define a local constant. - -### `cmd/obol/sell.go` — `sellPricingCommand` - -- Auto-discover wallet via `openclaw.ResolveWalletAddress(cfg)` when `--wallet` not set -- Remove `Required: true` from `--wallet` -- Update chain usage help: `"Payment chain (base-sepolia, base, ethereum)"` - -### Files -- `cmd/obol/sell.go` (`resolveX402Chain`, `sellPricingCommand`) -- `internal/x402/config.go` (`ResolveChain` — add ethereum) -- `internal/x402/config_test.go` (add ethereum test cases) -- `cmd/obol/sell_test.go` (update `TestResolveX402Chain`) - ---- - -## Phase 7: Tests & Docs - -### Tests -- `internal/erc8004/networks_test.go`: `ResolveNetwork` all chains, `ResolveNetworks` CSV parsing -- `internal/erc8004/signer_test.go`: HTTP mock for remote-signer API -- `internal/erc8004/sponsor_test.go`: EIP-712 construction, HTTP mock -- `internal/openclaw/wallet_resolve_test.go`: 0/1/multi instance -- `cmd/obol/sell_test.go`: Updated register flags, multi-chain parsing, new x402 chains - -### Docs -- `CLAUDE.md`: Update CLI command table, add `--chain` multi-value, remove `--rpc-url` -- `internal/embed/skills/sell/SKILL.md`: New registration flow, multi-network, remote-signer -- `internal/embed/skills/discovery/SKILL.md`: Multi-network registry info -- `cmd/obol/main.go`: Update root help text for sell register - ---- - -## Dependency Graph - -``` -Phase 1 (multi-network config) - ├──→ Phase 2 (remote-signer integration + registration flows) - └──→ Phase 3 (wallet auto-discovery) - │ - v - Phase 4 (rewrite sell register) ← depends on 1+2+3 - │ - v - Phase 5 (interactive prompts) ← depends on 3 (wallet discovery) - │ - v - Phase 6 (x402 payment chains + sell pricing) - │ - v - Phase 7 (tests & docs — throughout) -``` - ---- - -## Key Design Decisions - -1. **Remote-signer for all signing** — Never extract private keys. Use `POST /api/v1/sign/{address}/transaction` for direct registration, `POST /api/v1/sign/{address}/typed-data` for sponsored EIP-712. Access via temporary port-forward. - -2. **Local eRPC for all chain access** — `http://localhost/rpc/{network}` via k3d port mapping. No public RPCs. eRPC already has upstreams for mainnet, base, base-sepolia. - -3. **Multi-chain `--chain mainnet,base`** — Same agentURI and wallet registered on each chain. Best-effort: if one fails, continue to next. Update `registrations[]` array in `agent-registration.json` with all successes. - -4. **Prefer remote-signer, fallback to `--private-key-file`** — Auto-discover wallet → use remote-signer. If no instance found, accept `--private-key-file` for standalone usage. - -5. **Good defaults for registration metadata** — Pre-fill name (`Obol Agent`), description, image URL. Interactive mode lets users confirm or override each. - -6. **`charmbracelet/huh` for prompts** — Modern TUI with select, input, confirm. TTY-gated. - ---- - -## Key Files Summary - -| File | Change | -|------|--------| -| `internal/erc8004/networks.go` | New — multi-network config registry | -| `internal/erc8004/signer.go` | New — remote-signer REST API client | -| `internal/erc8004/register.go` | New — direct + sponsored registration flows | -| `internal/erc8004/sponsor.go` | New — sponsored API client | -| `internal/erc8004/client.go` | Add `NewClientForNetwork` | -| `internal/openclaw/wallet_resolve.go` | New — wallet address + namespace discovery | -| `cmd/obol/sell.go` | Rewrite register, add prompts to inference/http/register/pricing | -| `cmd/obol/main.go` | Wire `*ui.UI`, update help text | -| `cmd/obol/sell_test.go` | Update all affected tests | -| `internal/x402/config.go` | Add ethereum mainnet chain | - ---- - -## Verification - -```bash -# Phase 1 -go test ./internal/erc8004/ -run TestResolveNetwork - -# Phase 2 (unit — mock remote-signer) -go test ./internal/erc8004/ -run TestRemoteSigner -go test ./internal/erc8004/ -run TestSponsored - -# Phase 3 -go test ./internal/openclaw/ -run TestResolveWallet - -# Phase 4+5 (manual — needs running cluster + tunnel) -obol sell register --chain base-sepolia # direct tx via remote-signer -obol sell register --chain mainnet --sponsored # zero-gas via howto8004 -obol sell register --chain mainnet,base # multi-chain best-effort -obol sell inference # interactive prompts -obol sell http # interactive prompts -obol sell register # interactive with defaults to confirm - -# Phase 6 -obol sell pricing --chain base # auto-discovers wallet - -# All unit tests -go test ./cmd/obol/ -run TestSell -go test ./internal/erc8004/ -go test ./internal/openclaw/ -run TestResolve -go test ./internal/x402/ -run TestResolveChain -``` diff --git a/docs/plans/per-token-metering.md b/docs/plans/per-token-metering.md deleted file mode 100644 index a5839286..00000000 --- a/docs/plans/per-token-metering.md +++ /dev/null @@ -1,164 +0,0 @@ -# Per-Token Metering Plan - -## Scope - -This document defines phase 2 of issue 258: exact seller-side token metering -for paid inference offers, with Prometheus-native monitoring and a lightweight -status surface on `ServiceOffer`. - -Phase 1 is already deployed separately: - -- `perMTok` is accepted by the sell flows -- the enforced x402 charge is approximated as `perMTok / 1000` -- the source pricing metadata is persisted on each pricing route -- buyer and verifier expose operational Prometheus metrics - -This document covers how to replace that approximation for non-streaming -OpenAI-compatible chat completions. - -## Goals - -- Meter actual prompt, completion, and total token usage for paid inference - routes. -- Convert measured usage into estimated USDC using the seller's `perMTok`. -- Expose seller-side metrics through Prometheus. -- Surface roll-up usage on `ServiceOffer.status.usage`. -- Keep the verifier as the pre-request payment gate. - -## Non-Goals - -- Post-pay settlement or escrow. -- Exact metering for streaming responses. -- Exact metering for non-OpenAI response formats. -- Buyer-side billing authority. Buyer token telemetry remains observational. - -## Request Flow - -```text -client - -> Traefik HTTPRoute - -> x402-verifier (pre-request payment gate) - -> x402-meter - -> upstream inference service - -> x402-meter parses usage.total_tokens - -> response returned to client - -> x402-meter exports Prometheus metrics and updates ServiceOffer.status.usage -``` - -Key point: - -- `x402-verifier` still decides whether a request may proceed. -- `x402-meter` becomes the source of truth for exact usage accounting after the - upstream response is known. - -## Config Schema - -`x402-meter` is configured per monetized route. - -```yaml -apiVersion: v1 -kind: ConfigMap -metadata: - name: x402-meter-config - namespace: x402 -data: - config.yaml: | - routes: - - pattern: /services/my-qwen/v1/chat/completions - offerNamespace: llm - offerName: my-qwen - upstreamURL: http://ollama.llm.svc.cluster.local:11434 - upstreamAuth: "" - perMTok: "1.25" - priceModel: perMTok - responseFormat: openai-chat-completions -``` - -Required fields: - -- `pattern` -- `offerNamespace` -- `offerName` -- `upstreamURL` -- `perMTok` - -Optional fields: - -- `upstreamAuth` -- `responseFormat` - -## Status Schema - -`ServiceOffer.status.usage` is extended with a seller-side rollup: - -```yaml -status: - usage: - requests: 124 - promptTokens: 102400 - completionTokens: 18432 - totalTokens: 120832 - estimatedUSDC: "0.15104" - lastUpdated: "2026-03-06T12:34:56Z" -``` - -Rules: - -- `estimatedUSDC` is derived from `totalTokens / 1_000_000 * perMTok` -- values are monotonic rollups, not per-request histories -- writes should be batched to avoid excessive CR status churn - -## Prometheus Metrics - -`x402-meter` exposes `/metrics` and is scraped through a `ServiceMonitor`. - -Metric set: - -- `obol_x402_meter_requests_total{offer_namespace,offer_name,route}` -- `obol_x402_meter_prompt_tokens_total{offer_namespace,offer_name,route}` -- `obol_x402_meter_completion_tokens_total{offer_namespace,offer_name,route}` -- `obol_x402_meter_total_tokens_total{offer_namespace,offer_name,route}` -- `obol_x402_meter_estimated_usdc_total{offer_namespace,offer_name,route}` -- `obol_x402_meter_parse_failures_total{offer_namespace,offer_name,route}` - -Label guidance: - -- keep labels limited to offer identity and route pattern -- do not label by user, wallet, or request id - -## Buyer-Side Observational Metrics - -Buyer-side metrics remain separate from billing: - -- `x402-buyer` continues exposing request, payment, auth pool, and active model - mapping metrics. -- A later extension may parse `usage.total_tokens` from remote seller responses - and emit observational counters keyed by `upstream` and `remote_model`. -- Disagreement between buyer-observed tokens and seller-billed tokens should be - treated as an alerting or debugging signal, not a settlement input. - -## Rollout Plan - -1. Deploy `x402-meter` behind the verifier for one non-streaming paid route. -2. Validate token parsing and Prometheus scrape health. -3. Enable `ServiceOffer.status.usage` updates with rate limiting. -4. Switch sell-side status output from approximation-first to exact-usage-first - whenever meter data is present. -5. Keep the phase-1 `perMTok / 1000` approximation as a fallback for routes not - yet migrated to `x402-meter`. - -## Failure Handling - -- If the response body cannot be parsed, increment - `obol_x402_meter_parse_failures_total` and return the upstream response - unchanged. -- If the upstream omits `usage.total_tokens`, do not synthesize exact billing. -- If status updates fail, metrics must still be emitted. -- If Prometheus is unavailable, request serving must continue. - -## Open Questions - -- Whether streamed responses should be handled with token trailers, chunk - aggregation, or remain explicitly unsupported. -- Whether meter state should be derived solely from Prometheus counters or also - persisted locally for faster CR status reconciliation. diff --git a/docs/pr-review-monetize-path.md b/docs/pr-review-monetize-path.md deleted file mode 100644 index 7afeaef8..00000000 --- a/docs/pr-review-monetize-path.md +++ /dev/null @@ -1,117 +0,0 @@ -# PR Review Transcript: `feat/monetize-path` - -**Reviewer**: Oisin + Claude Code | **Date**: 2026-04-07 | **Branch**: `feat/monetize-path` into `main` - -212 files changed, ~29K insertions. Bulk of the diff is the new serviceoffer-controller (Go), autoresearch skills, flow validation scripts, and lint/cosmetic cleanup. - ---- - -## 1. Reth ERC-8004 Indexer — Removed - -**Finding**: A standalone Rust crate (`reth-erc8004-indexer/`) and `Dockerfile.reth-erc8004-indexer` were added but had zero consumers. Discovery uses on-chain RPC; autoresearch coordinator hardcodes 8004scan.io. The `internal/network/validate.go` had config validation for indexer flags but `validateInstallOptions()` was never called from production code, and the `values.yaml.gotmpl` was never updated with the template fields. - -**Decision**: Removed entirely. The ExEx-on-archive-reth concept is sound — when a consumer needs indexed ERC-8004 data, it can be brought back behind an `--archive` flag on `obol network install ethereum` (gated to reth-only, piggybacking on reth's execution extension system). - -**Removed**: -- `reth-erc8004-indexer/` (entire Rust crate) -- `Dockerfile.reth-erc8004-indexer` -- `internal/network/validate.go` + `validate_test.go` (dead code) -- `ralph-m3.md` (Codex prompt for indexer validation) -- Indexer references in `autoresearch-coordinator/SKILL.md` (`OBOL_INDEXER_API_URL` env var, "internal indexer preference" language) - ---- - -## 2. Significant Non-Cosmetic Changes Identified - -| Change | Scope | -|--------|-------| -| **serviceoffer-controller** (Go) | ~2400 lines. Replaces Python `monetize.py` reconciler with a proper K8s controller watching ServiceOffers, creating child resources (Middleware, HTTPRoute, RegistrationRequest). | -| **RegistrationRequest CRD** | New resource decoupling ERC-8004 registration from ServiceOffer lifecycle. | -| **x402 verifier ServiceOffer source** | Verifier now watches ServiceOffers directly for live pricing routes via `WatchServiceOffers()` + `ConfigAccumulator`, instead of relying solely on the static `x402-pricing` ConfigMap. | -| **`monetize.py` rewrite** | Gutted to a compatibility shim — creates/deletes ServiceOffers via K8s API, waits for controller convergence. `/skill.md` publishing is a no-op. | -| **Autoresearch skills** | 3 new skills (coordinator, worker, autoresearch) with `Dockerfile.worker`. Deployment path is agent-driven via the skill + `obol sell http`. | -| **`flows/`** | 10 end-to-end validation scripts covering the full user journey. | -| **dev_registry.go** | Local k3d registry mirrors for docker.io/ghcr.io/quay.io. | - ---- - -## 3. x402 ConfigMap vs ServiceOffer Informer — Tech Debt Assessment - -**Question**: Is the dual route source (ConfigMap file watcher + ServiceOffer informer) tech debt? - -**Answer**: Partially. The `x402-pricing` ConfigMap is still needed for **global identity** (wallet, chain, facilitatorURL) — the verifier needs these at startup before any ServiceOffer exists. However, the `routes[]` array in the ConfigMap is vestigial. Nothing in the current flow writes routes to it; the controller creates routes via the informer path. `AddRoute()`, `DeleteStaticOfferRoute()`, `DeletePaymentRoute()`, `WritePricingConfig()`, and all `RouteOption` types were dead code — only called from one integration test. - -**Action**: Removed all dead route-management functions from `setup.go`. Replaced the integration test's `addPricingRoute()` helper with a no-op + sleep (controller handles routes now). Cleaned up `setup_test.go` to remove tests for deleted functions. Updated `docs/x402-test-plan.md`. - ---- - -## 4. Bugs Fixed - -### Critical -| Bug | Location | Fix | -|-----|----------|-----| -| **Silent tombstone failure** — `SetMetadata()` error discarded with `_` on the on-chain deactivation call. Deleted service stays registered as active. | `controller.go:778` | Log the error so operators know metadata wasn't cleared. | -| **Port parsing ignores errors** — `strconv.Atoi(port)` error discarded, malformed port silently becomes 0, creating broken K8s Services. | `sell.go:1931, 2012` | Validate and return user-facing error. Changed `buildInferenceServiceOfferSpec` to return `(map, error)`. | -| **Deletion requeue bug** — `reconcileDeletingOffer()` returns `nil` when cleanup isn't ready, causing `Forget()` on the workqueue. Offer stalls in deleting state. | `controller.go:366` | Return error to trigger `AddRateLimited` requeue. | - -### High -| Bug | Location | Fix | -|-----|----------|-----| -| **CRD port field unvalidated** — no min/max, user could set port 0 or 99999. | `serviceoffer-crd.yaml:102` | Added `minimum: 1, maximum: 65535`. | -| **CRD path field unvalidated** — user could set `/../admin` or empty string. | `serviceoffer-crd.yaml:182` | Added `pattern: "^/[a-zA-Z0-9/_.-]*$"`. | -| **Probe URL unsanitized** — endpoint from CRD metadata concatenated directly into URL. | `sell.go:998` | Validate endpoint starts with `/`, reject `..` traversal. | -| **K8s name overflow** — `childName("so-" + name)` had no length check. Long ServiceOffer names exceed 253-char DNS limit. | `render.go:445` | Added `safeName()` helper with hash-based truncation. New tests in `render_test.go`. | - -### Medium -| Bug | Location | Fix | -|-----|----------|-----| -| **`/skill.md` doc mismatch** — `monetize.py` disables publishing (no-op) but `SKILL.md` documents it as active. | `sell/SKILL.md:110` | Updated architecture diagram to reflect controller ownership. | -| **Metadata sync not surfaced** — error logged but status set to "Registered" anyway. | `controller.go:680` | Improved logging with explicit success/failure messages. | -| **No log on key loading** — silent whether ERC-8004 signing key loaded or not. | `controller.go:82` | Added startup log indicating key presence or absence. | - ---- - -## 5. Dead Code Removed - -| Item | Reason | -|------|--------| -| `sellInfoCommand` — defined but not registered | Wired it up (user's choice to keep it). | -| `AddRoute()`, `RouteOption`, `With*` options, `DeleteStaticOfferRoute`, `DeletePaymentRoute`, `WritePricingConfig`, `sameRouteIdentity` | ConfigMap route path replaced by ServiceOffer informer. | -| `ralph-m1.md`, `ralph-m2.md`, `ralph-m3.md` | Codex agent prompt files, not project artifacts. | - ---- - -## 6. Rename: `sell probe` to `sell test` - -Renamed `obol sell probe` to `obol sell test` with a simplified user-facing description ("Test that a service is live and requiring payment"). Updated across: -- `cmd/obol/sell.go` (command name, usage, description, examples, error messages) -- `internal/embed/skills/monetize-guide/SKILL.md` -- `internal/embed/skills/monetize-guide/references/seller-prompt.md` - ---- - -## 7. Deferred / Not Addressed - -| Item | Reason | -|------|--------| -| **Controller RBAC too broad** — cluster-wide Services/ConfigMaps/Deployments mutation. | Intentionally deferred; tightening namespace scope is a follow-up. | -| **Test coverage gaps** — controller has 101 lines of tests for 1225 LOC; sell_test.go tests structure only, not actions. | Flagged, not blocking. Controller needs integration tests before handling real money. | -| **Dual creation path** — both Go CLI and monetize.py can create ServiceOffers without coordination. | Architectural debt from the migration. monetize.py is documented as a compatibility shim. | - ---- - -## 8. Retracted Concerns - -| Initial Concern | Why Retracted | -|----------------|---------------| -| Race condition in `registrationOwner()` | Informer store (`cache.Store`) is thread-safe — uses internal locking. | -| `Dockerfile.worker` has no deployment path | It does — via the `autoresearch-worker` skill + `obol sell http`. Agent-driven, not CLI-driven. | - ---- - -## 9. Verification - -All changes verified: -- `go build ./...` — clean -- `go test ./...` — all 22 test packages pass, zero failures -- `grep` sweep — no stale references to removed code diff --git a/docs/x402-test-plan.md b/docs/x402-test-plan.md deleted file mode 100644 index ac843764..00000000 --- a/docs/x402-test-plan.md +++ /dev/null @@ -1,333 +0,0 @@ -# x402 + ERC-8004 Integration Test Plan - -**Feature branch:** `feat/secure-enclave-inference` -**Scope:** 100% coverage of x402 payment gating, ERC-8004 on-chain registration, verifier service, and CLI commands. - -> Historical note: `/.well-known/agent-registration.json` is no longer served by `x402-verifier`. -> Registration publication now belongs to `serviceoffer-controller` and `RegistrationRequest`, so any verifier-specific well-known endpoint references below are outdated. - ---- - -## 1. Coverage Inventory - -### Current State - -| Package | File | Existing Tests | Coverage | -|---------|------|---------------|----------| -| `internal/erc8004` | `client.go` | TestNewClient, TestRegister | ~60% (missing SetAgentURI, SetMetadata error paths) | -| `internal/erc8004` | `store.go` | TestStore | ~70% (missing Save errors, corrupt file) | -| `internal/erc8004` | `types.go` | none | 0% (JSON marshaling/unmarshaling) | -| `internal/erc8004` | `abi.go` | implicit via client tests | ~50% (missing ABI parse error, constant verification) | -| `internal/x402` | `verifier.go` | 11 tests | ~85% (missing SetRegistration, HandleWellKnown) | -| `internal/x402` | `matcher.go` | 8 tests | ~95% (good) | -| `internal/x402` | `config.go` | implicit via verifier | ~40% (missing LoadConfig, ResolveChain edge cases) | -| `internal/x402` | `watcher.go` | none | 0% | -| `internal/x402` | `setup.go` | none | 0% (kubectl-dependent, needs mock) | -| `cmd/obol` | `monetize.go` | none | 0% | - -### Target: 100% Function Coverage - ---- - -## 2. Unit Tests to Add - -### 2.1 `internal/erc8004` Package - -#### `abi_test.go` (NEW) - -| Test | What it verifies | Priority | -|------|-----------------|----------| -| `TestABI_ParsesSuccessfully` | Embedded ABI JSON parses without error | HIGH | -| `TestABI_AllFunctionsPresent` | All 10 functions present: register (3 overloads), setAgentURI, setMetadata, getMetadata, getAgentWallet, setAgentWallet, unsetAgentWallet, tokenURI | HIGH | -| `TestABI_AllEventsPresent` | All 3 events present: Registered, URIUpdated, MetadataSet | HIGH | -| `TestABI_RegisterOverloads` | 3 distinct register methods exist with correct input counts (0, 1, 2) | HIGH | -| `TestConstants_Addresses` | IdentityRegistryBaseSepolia, ReputationRegistryBaseSepolia, ValidationRegistryBaseSepolia are valid hex addresses (40 chars after 0x) | MEDIUM | -| `TestConstants_ChainID` | BaseSepoliaChainID == 84532 | LOW | - -#### `types_test.go` (NEW) - -| Test | What it verifies | Priority | -|------|-----------------|----------| -| `TestAgentRegistration_MarshalJSON` | Full struct serializes to spec-compliant JSON (type, name, description, image, services, x402Support, active, registrations, supportedTrust) | HIGH | -| `TestAgentRegistration_UnmarshalJSON` | Canonical spec JSON (from ERC8004SPEC.md) deserializes correctly | HIGH | -| `TestAgentRegistration_OmitEmptyFields` | Optional fields (description, image, registrations, supportedTrust) omitted when zero-value | MEDIUM | -| `TestServiceDef_VersionOptional` | ServiceDef without version marshals correctly (version omitempty) | MEDIUM | -| `TestOnChainReg_AgentIDNumeric` | AgentID is int64, serializes as JSON number (not string) | HIGH | -| `TestRegistrationType_Constant` | RegistrationType == `"https://eips.ethereum.org/EIPS/eip-8004#registration-v1"` | LOW | - -#### `client_test.go` (ADDITIONS to existing) - -| Test | What it verifies | Priority | -|------|-----------------|----------| -| `TestNewClient_DialError` | Returns error when RPC URL is unreachable | MEDIUM | -| `TestNewClient_ChainIDError` | Returns error when eth_chainId fails | MEDIUM | -| `TestSetAgentURI` | Successful tx + wait mined (mock sendRawTransaction + receipt) | HIGH | -| `TestSetMetadata` | Successful tx + wait mined | HIGH | -| `TestRegister_NoRegisteredEvent` | Returns error when receipt has no Registered event log | HIGH | -| `TestRegister_TxError` | Returns error when sendRawTransaction fails | MEDIUM | -| `TestGetMetadata_EmptyResult` | Returns nil when contract returns empty bytes | MEDIUM | - -#### `store_test.go` (ADDITIONS to existing) - -| Test | What it verifies | Priority | -|------|-----------------|----------| -| `TestStore_SaveOverwrite` | Second Save overwrites first | MEDIUM | -| `TestStore_LoadCorruptJSON` | Returns error on malformed JSON file | MEDIUM | -| `TestStore_SaveReadOnly` | Returns error when directory is read-only (permission denied) | LOW | - -### 2.2 `internal/x402` Package - -#### `verifier_test.go` (ADDITIONS) - -| Test | What it verifies | Priority | -|------|-----------------|----------| -| `TestVerifier_SetRegistration` | SetRegistration stores data, HandleWellKnown returns it | HIGH | -| `TestVerifier_HandleWellKnown_NoRegistration` | Returns 404 when no registration set | HIGH | -| `TestVerifier_HandleWellKnown_JSON` | Response is valid JSON AgentRegistration with correct Content-Type | HIGH | -| `TestVerifier_ReadyzNotReady` | Returns 503 when config is nil (fresh Verifier without config) | MEDIUM | - -#### `config_test.go` (NEW) - -| Test | What it verifies | Priority | -|------|-----------------|----------| -| `TestLoadConfig_ValidYAML` | Parses complete YAML with wallet, chain, routes | HIGH | -| `TestLoadConfig_Defaults` | Empty chain defaults to "base", empty facilitatorURL defaults | HIGH | -| `TestLoadConfig_InvalidYAML` | Returns parse error on malformed YAML | MEDIUM | -| `TestLoadConfig_FileNotFound` | Returns read error | MEDIUM | -| `TestResolveChain_AllSupported` | All 6 chain names resolve (base, base-sepolia, polygon, polygon-amoy, avalanche, avalanche-fuji) | HIGH | -| `TestResolveChain_Aliases` | "base-mainnet" == "base", "polygon-mainnet" == "polygon", etc. | MEDIUM | -| `TestResolveChain_Unsupported` | Returns error for unknown chain name | MEDIUM | -| `TestResolveChain_ErrorMessage` | Error message lists all supported chains | LOW | - -#### `watcher_test.go` (NEW) - -| Test | What it verifies | Priority | -|------|-----------------|----------| -| `TestWatchConfig_DetectsChange` | Write new config file, watcher reloads verifier within interval | HIGH | -| `TestWatchConfig_IgnoresUnchanged` | Same mtime = no reload | MEDIUM | -| `TestWatchConfig_InvalidConfig` | Bad YAML doesn't crash watcher, verifier keeps old config | HIGH | -| `TestWatchConfig_CancelContext` | Context cancellation stops the watcher goroutine cleanly | MEDIUM | -| `TestWatchConfig_MissingFile` | Missing file logged but watcher continues | MEDIUM | - -#### `setup_test.go` (NEW — requires abstraction for kubectl) - -The `setup.go` file shells out to `kubectl`. To unit-test it, extract an interface: - -```go -// KubectlRunner abstracts kubectl execution for testing. -type KubectlRunner interface { - Run(args ...string) error - Output(args ...string) (string, error) -} -``` - -| Test | What it verifies | Priority | -|------|-----------------|----------| -| `TestSetup_PatchesSecretAndConfigMap` | Calls kubectl patch on both secret and configmap with correct args | HIGH | -| `TestSetup_NoKubeconfig` | Returns "cluster not running" error | HIGH | -| `TestAddRoute_AppendsToExisting` | Reads existing config, appends route, patches back | HIGH | -| `TestAddRoute_FirstRoute` | Adds route when routes list is empty | MEDIUM | -| `TestGetPricingConfig_EmptyResponse` | Returns empty PricingConfig when configmap has no data | MEDIUM | -| `TestGetPricingConfig_ParsesYAML` | Correct wallet/chain/routes from kubectl output | HIGH | -| `TestPatchPricingConfig_Serialization` | Generated YAML has correct format (routes array, descriptions) | MEDIUM | - ---- - -## 3. Integration Tests (//go:build integration) - -These require a running k3d cluster with `OBOL_DEVELOPMENT=true`. - -### 3.1 `internal/x402/integration_test.go` (NEW) - -**Prerequisites:** Running cluster, x402 namespace deployed. - -| Test | What it verifies | Runtime | Priority | -|------|-----------------|---------|----------| -| `TestIntegration_X402Setup` | `obol x402 setup --wallet 0x... --chain base-sepolia` patches configmap + secret in cluster | 30s | HIGH | -| `TestIntegration_X402Status` | `obol x402 status` reads correct config from cluster | 15s | HIGH | -| `TestIntegration_X402ServiceOfferRoutes` | Create ServiceOffer, verify verifier picks up live route via informer | 30s | MEDIUM | -| `TestIntegration_VerifierDeployment` | x402-verifier pod is running, responds to /healthz | 15s | HIGH | -| `TestIntegration_VerifierForwardAuth` | Send request to /verify endpoint with X-Forwarded-Uri, verify 200/402 behavior | 30s | HIGH | -| `TestIntegration_WellKnownEndpoint` | GET /.well-known/agent-registration.json returns valid JSON (after registration set) | 15s | MEDIUM | - -### 3.2 `internal/erc8004/integration_test.go` (NEW) - -**Prerequisites:** Base Sepolia RPC access, funded test wallet (ERC8004_PRIVATE_KEY env var). - -| Test | What it verifies | Runtime | Priority | -|------|-----------------|---------|----------| -| `TestIntegration_RegisterOnBaseSepolia` | Full register() tx on testnet, verify agentID returned | 60s | HIGH | -| `TestIntegration_SetAgentURI` | setAgentURI() after register, verify tokenURI() returns new URI | 60s | HIGH | -| `TestIntegration_SetAndGetMetadata` | setMetadata() + getMetadata() roundtrip | 60s | MEDIUM | -| `TestIntegration_GetAgentWallet` | getAgentWallet() returns owner address after registration | 30s | MEDIUM | - -**Skip logic:** -```go -func TestMain(m *testing.M) { - if os.Getenv("ERC8004_PRIVATE_KEY") == "" { - fmt.Println("Skipping ERC-8004 integration tests: ERC8004_PRIVATE_KEY not set") - os.Exit(0) - } - os.Exit(m.Run()) -} -``` - -### 3.3 End-to-End: x402 Payment Flow - -**File:** `internal/x402/e2e_test.go` (NEW, `//go:build integration`) - -**Prerequisites:** Running cluster with inference network deployed, x402 enabled, funded test wallet. - -| Test | Scenario | Steps | Priority | -|------|----------|-------|----------| -| `TestE2E_InferenceWithPayment` | Full x402 payment lifecycle | 1. Deploy inference network with x402Enabled=true; 2. Configure pricing via AddRoute; 3. Send request WITHOUT payment → 402; 4. Verify 402 body contains payment requirements; 5. Send request WITH valid x402 payment header → 200 | HIGH | -| `TestE2E_RegisterAndServeWellKnown` | ERC-8004 + well-known endpoint | 1. Register agent on Base Sepolia; 2. Set registration on verifier; 3. GET /.well-known/agent-registration.json → matches registration | MEDIUM | - ---- - -## 4. CLI Command Tests - -### `cmd/obol/x402_test.go` (NEW) - -Pattern: Build the CLI app, run subcommands against mocked infrastructure. - -| Test | What it verifies | Priority | -|------|-----------------|----------| -| `TestX402Command_Structure` | x402 has 3 subcommands: register, setup, status | HIGH | -| `TestX402Register_RequiresPrivateKey` | Fails without --private-key or ERC8004_PRIVATE_KEY | HIGH | -| `TestX402Register_TrimsHexPrefix` | 0x-prefixed key handled correctly | MEDIUM | -| `TestX402Setup_RequiresWallet` | Fails without --wallet flag | HIGH | -| `TestX402Setup_DefaultChain` | Default chain is "base" | MEDIUM | -| `TestX402Status_NoCluster` | Graceful output when no cluster running | MEDIUM | -| `TestX402Status_NoRegistration` | Shows "not registered" message | MEDIUM | - ---- - -## 5. Helmfile Template Tests - -### Infrastructure Helmfile (conditional x402 resources) - -**File:** `internal/embed/infrastructure/helmfile_test.go` (NEW) - -| Test | What it verifies | Priority | -|------|-----------------|----------| -| `TestHelmfile_X402DisabledByDefault` | x402Enabled=false: no Middleware CRD rendered, no ExtensionRef on eRPC HTTPRoute | HIGH | -| `TestHelmfile_X402Enabled` | x402Enabled=true: Middleware CRD rendered with correct ForwardAuth address, ExtensionRef added to eRPC HTTPRoute | HIGH | - -### Inference Network Template - -**File:** `internal/embed/networks/inference/template_test.go` (NEW) - -| Test | What it verifies | Priority | -|------|-----------------|----------| -| `TestInferenceValues_X402EnabledField` | values.yaml.gotmpl contains x402Enabled field with @enum true,false, @default false | HIGH | -| `TestInferenceHelmfile_X402Passthrough` | x402Enabled value passed through to helmfile.yaml.gotmpl | HIGH | -| `TestInferenceGateway_ConditionalMiddleware` | gateway.yaml: Middleware CRD only rendered when x402Enabled=true | HIGH | -| `TestInferenceGateway_ConditionalExtensionRef` | gateway.yaml: ExtensionRef only present when x402Enabled=true | HIGH | - ---- - -## 6. Coverage Gap Analysis — Functions NOT Tested - -### internal/erc8004 - -| Function | File:Line | Test Status | Action | -|----------|-----------|-------------|--------| -| `NewClient()` | client.go:26 | TESTED | - | -| `Close()` | client.go:57 | implicit | - | -| `Register()` | client.go:63 | TESTED | Add error paths | -| `SetAgentURI()` | client.go:95 | **UNTESTED** | Add test | -| `SetMetadata()` | client.go:114 | **UNTESTED** | Add test | -| `GetMetadata()` | client.go:133 | TESTED | Add empty result | -| `TokenURI()` | client.go:150 | TESTED | - | -| `NewStore()` | store.go:30 | implicit | - | -| `Save()` | store.go:39 | TESTED | Add error paths | -| `Load()` | store.go:55 | TESTED | Add corrupt file | - -### internal/x402 - -| Function | File:Line | Test Status | Action | -|----------|-----------|-------------|--------| -| `NewVerifier()` | verifier.go:25 | TESTED | - | -| `Reload()` | verifier.go:34 | TESTED | - | -| `HandleVerify()` | verifier.go:56 | TESTED (11 cases) | - | -| `HandleHealthz()` | verifier.go:114 | TESTED | - | -| `HandleReadyz()` | verifier.go:120 | TESTED | Add nil config case | -| `SetRegistration()` | verifier.go:131 | **UNTESTED** | Add test | -| `HandleWellKnown()` | verifier.go:136 | **UNTESTED** | Add test | -| `LoadConfig()` | config.go:46 | **UNTESTED** | Add tests | -| `ResolveChain()` | config.go:69 | partial (error case only) | Add all chains | -| `WatchConfig()` | watcher.go:16 | **UNTESTED** | Add tests | -| `Setup()` | setup.go:23 | **UNTESTED** | Needs kubectl abstraction | -| `WatchServiceOffers()` | serviceoffer_source.go:23 | **UNTESTED** | Integration only | -| `GetPricingConfig()` | setup.go:96 | **UNTESTED** | Needs kubectl abstraction | -| `matchRoute()` | matcher.go:19 | TESTED (8 cases) | - | -| `matchPattern()` | matcher.go:29 | TESTED | - | -| `globMatch()` | matcher.go:52 | TESTED | - | - ---- - -## 7. Implementation Priority - -### Phase 1: Unit tests (no cluster needed) — ~2 hours - -1. `internal/erc8004/abi_test.go` — ABI integrity checks -2. `internal/erc8004/types_test.go` — JSON serialization spec compliance -3. `internal/x402/config_test.go` — LoadConfig + ResolveChain -4. `internal/x402/verifier_test.go` — SetRegistration + HandleWellKnown additions -5. `internal/x402/watcher_test.go` — File watcher - -### Phase 2: Missing client methods + error paths — ~1 hour - -6. `internal/erc8004/client_test.go` — SetAgentURI, SetMetadata, error paths -7. `internal/erc8004/store_test.go` — overwrite, corrupt, permissions - -### Phase 3: Setup abstraction + tests — ~1.5 hours - -8. Extract `KubectlRunner` interface from `setup.go` -9. `internal/x402/setup_test.go` — all Setup/AddRoute/GetPricingConfig - -### Phase 4: Integration tests — ~2 hours (requires running cluster) - -10. `internal/x402/integration_test.go` — cluster-based tests -11. `internal/erc8004/integration_test.go` — Base Sepolia testnet tests - -### Phase 5: Template + CLI tests — ~1 hour - -12. Helmfile template rendering tests -13. `cmd/obol/x402_test.go` — CLI command structure + validation - ---- - -## 8. Test Execution Commands - -```bash -# Phase 1-3: Unit tests only -go test -v ./internal/erc8004/... ./internal/x402/... - -# Phase 4: Integration tests (requires cluster + testnet key) -export OBOL_CONFIG_DIR=$(pwd)/.workspace/config -export OBOL_BIN_DIR=$(pwd)/.workspace/bin -export OBOL_DATA_DIR=$(pwd)/.workspace/data -export ERC8004_PRIVATE_KEY= -go build -o .workspace/bin/obol ./cmd/obol -go test -tags integration -v -timeout 15m ./internal/x402/ ./internal/erc8004/ - -# Coverage report -go test -coverprofile=coverage.out ./internal/erc8004/... ./internal/x402/... -go tool cover -html=coverage.out -o coverage.html -``` - ---- - -## 9. Success Criteria - -- [ ] 100% function coverage on `internal/erc8004/` (all 10 exported functions) -- [ ] 100% function coverage on `internal/x402/` (all 14 exported functions) -- [ ] All 3 ABI register overloads verified against canonical ABI -- [ ] JSON serialization roundtrip matches ERC-8004 spec format -- [ ] WatchConfig tested with file changes, cancellation, and error recovery -- [ ] Setup/AddRoute/GetPricingConfig tested via kubectl mock -- [ ] HandleWellKnown tested (200 with data, 404 without) -- [ ] Integration tests skip gracefully when prerequisites unavailable -- [ ] `go test ./...` passes with zero failures diff --git a/internal/embed/embed_skills_test.go b/internal/embed/embed_skills_test.go index 97ec76b4..fb09f116 100644 --- a/internal/embed/embed_skills_test.go +++ b/internal/embed/embed_skills_test.go @@ -93,6 +93,7 @@ func TestCopySkills(t *testing.T) { for _, sub := range []string{ "sell/scripts/monetize.py", "sell/references/serviceoffer-spec.md", + "sell/references/registrationrequest-spec.md", "sell/references/x402-pricing.md", } { if _, err := os.Stat(filepath.Join(destDir, sub)); err != nil { @@ -100,6 +101,16 @@ func TestCopySkills(t *testing.T) { } } + // buy-inference must have references/ + for _, sub := range []string{ + "buy-inference/references/purchase-request-spec.md", + "buy-inference/references/x402-buyer-api.md", + } { + if _, err := os.Stat(filepath.Join(destDir, sub)); err != nil { + t.Errorf("missing %s: %v", sub, err) + } + } + // discovery must have scripts/discovery.py and references/ for _, sub := range []string{ "discovery/scripts/discovery.py", diff --git a/internal/embed/skills/buy-inference/SKILL.md b/internal/embed/skills/buy-inference/SKILL.md index 561053dd..5103ee8a 100644 --- a/internal/embed/skills/buy-inference/SKILL.md +++ b/internal/embed/skills/buy-inference/SKILL.md @@ -205,5 +205,6 @@ python3 scripts/buy.py maintain ## References +- `references/purchase-request-spec.md` — Full `PurchaseRequest` CRD field reference - `references/x402-buyer-api.md` — Wire formats for 402 responses, X-PAYMENT headers, and sidecar config - See also: `discovery` skill for finding sellers on the ERC-8004 registry diff --git a/internal/embed/skills/buy-inference/references/purchase-request-spec.md b/internal/embed/skills/buy-inference/references/purchase-request-spec.md new file mode 100644 index 00000000..8019f888 --- /dev/null +++ b/internal/embed/skills/buy-inference/references/purchase-request-spec.md @@ -0,0 +1,113 @@ +# PurchaseRequest CRD Reference + +Group: `obol.org`, Version: `v1alpha1`, Kind: `PurchaseRequest` + +`PurchaseRequest` declares a remote x402-gated model that the buyer side of the +stack should fund and expose locally as `paid/`. The controller +turns the declared request into buyer config/auth material for the `x402-buyer` +sidecar in the `llm` namespace. + +## Example + +```yaml +apiVersion: obol.org/v1alpha1 +kind: PurchaseRequest +metadata: + name: remote-qwen + namespace: llm +spec: + endpoint: https://seller.example.com/services/qwen/v1/chat/completions + model: qwen3.5:32b + count: 100 + preSignedAuths: + - signature: "0x..." + from: "0xBuyer..." + to: "0xSeller..." + value: "1000" + validAfter: "0" + validBefore: "1744761600" + nonce: "0x1234" + autoRefill: + enabled: false + threshold: 10 + count: 50 + payment: + network: base + payTo: "0xSeller..." + price: "1000" + asset: "0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913" +status: + publicModel: paid/qwen3.5:32b + remaining: 87 + spent: 13 + totalSigned: 100 +``` + +## Spec Fields + +### Top-Level Fields + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `spec.endpoint` | string | Yes | Full x402-gated inference endpoint URL | +| `spec.model` | string | Yes | Remote model identifier exposed locally as `paid/` | +| `spec.count` | integer | Yes | Number of pre-signed auths to prepare | +| `spec.preSignedAuths` | array | No | ERC-3009 authorizations embedded by `buy.py` | +| `spec.autoRefill` | object | No | Future refill policy configuration | +| `spec.payment` | object | Yes | Seller payment requirements used for validation and routing | + +### `spec.preSignedAuths[]` + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `signature` | string | Yes | ERC-3009 signature | +| `from` | string | Yes | Buyer wallet address | +| `to` | string | Yes | Seller wallet address | +| `value` | string | Yes | Transfer amount in token base units | +| `validAfter` | string | Yes | Earliest validity timestamp | +| `validBefore` | string | Yes | Expiry timestamp | +| `nonce` | string | Yes | Single-use authorization nonce | + +### `spec.autoRefill` + +| Field | Type | Required | Default | Description | +|-------|------|----------|---------|-------------| +| `enabled` | boolean | No | `false` | Enables automatic refill behavior when implemented | +| `threshold` | integer | No | — | Refill when `remaining < threshold` | +| `count` | integer | No | — | Number of new auths to sign on refill | +| `maxTotal` | integer | No | — | Hard cap on total signed auths | +| `maxSpendPerDay` | string | No | — | Daily spend ceiling | + +### `spec.payment` + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `spec.payment.network` | string | Yes | Chain name for settlement | +| `spec.payment.payTo` | string | Yes | Seller recipient address | +| `spec.payment.price` | string | Yes | Per-request price in token base units | +| `spec.payment.asset` | string | Yes | Token contract address, typically USDC | + +## Status Fields + +| Field | Type | Description | +|-------|------|-------------| +| `status.observedGeneration` | integer | Last observed spec generation | +| `status.conditions` | array | Kubernetes-style conditions, including `Ready` | +| `status.publicModel` | string | LiteLLM model alias, usually `paid/` | +| `status.remaining` | integer | Remaining unused auths | +| `status.spent` | integer | Number of auths already consumed | +| `status.totalSigned` | integer | Total auths signed for the request | +| `status.totalSpent` | string | Total spend in token base units | +| `status.probedAt` | string | Probe timestamp | +| `status.probedPrice` | string | Price observed during probe | +| `status.walletBalance` | string | Buyer wallet balance at last reconciliation | +| `status.signerAddress` | string | Remote-signer address used for auth creation | + +## Lifecycle Notes + +- `buy.py buy` is the expected authoring path for this CRD. +- The controller validates the declared payment fields against the probed seller + endpoint before publishing the local paid route. +- Runtime spending is bounded by the number of embedded pre-signed auths. +- The controller writes the resulting buyer config/auth material for the + `x402-buyer` sidecar; the sidecar itself never gets signer access. diff --git a/internal/embed/skills/sell/SKILL.md b/internal/embed/skills/sell/SKILL.md index da0c39ec..bed4ab19 100644 --- a/internal/embed/skills/sell/SKILL.md +++ b/internal/embed/skills/sell/SKILL.md @@ -114,4 +114,5 @@ ServiceOffer CR (obol.org/v1alpha1) ## References - `references/serviceoffer-spec.md` — Full CRD field reference +- `references/registrationrequest-spec.md` — Child CRD used for publication and ERC-8004 side effects - `references/x402-pricing.md` — x402 pricing model details diff --git a/internal/embed/skills/sell/references/registrationrequest-spec.md b/internal/embed/skills/sell/references/registrationrequest-spec.md new file mode 100644 index 00000000..90a9ab6f --- /dev/null +++ b/internal/embed/skills/sell/references/registrationrequest-spec.md @@ -0,0 +1,59 @@ +# RegistrationRequest CRD Reference + +Group: `obol.org`, Version: `v1alpha1`, Kind: `RegistrationRequest` + +`RegistrationRequest` isolates registration publication and ERC-8004 side +effects from the main `ServiceOffer` reconciliation loop. `ServiceOffer` +remains the source of truth; this CRD exists so publication state can be +reconciled independently. + +## Example + +```yaml +apiVersion: obol.org/v1alpha1 +kind: RegistrationRequest +metadata: + name: so-qwen-inference + namespace: llm +spec: + serviceOfferName: qwen-inference + serviceOfferNamespace: llm + desiredState: Active +status: + phase: Published + publishedUrl: https://seller.example.com/.well-known/agent-registration.json + agentId: "1789" + registrationTxHash: "0xabc123..." + metadataSynced: true +``` + +## Spec Fields + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `spec.serviceOfferName` | string | Yes | Name of the parent `ServiceOffer` | +| `spec.serviceOfferNamespace` | string | Yes | Namespace of the parent `ServiceOffer` | +| `spec.desiredState` | string | Yes | Target publication state: `Active` or `Tombstoned` | + +## Status Fields + +| Field | Type | Description | +|-------|------|-------------| +| `status.phase` | string | Current reconciliation phase | +| `status.message` | string | Human-readable status detail | +| `status.publishedUrl` | string | URL where registration JSON was published | +| `status.agentId` | string | ERC-8004 token ID | +| `status.registrationTxHash` | string | Registration or tombstone transaction hash | +| `status.registrationOwner` | string | On-chain owner address for the registration | +| `status.registrationUri` | string | Published registration URI | +| `status.registrationSearchFromBlock` | integer | Block height hint for registration lookups | +| `status.metadataSynced` | boolean | Whether metadata mirroring completed successfully | + +## Lifecycle Notes + +- `desiredState: Active` publishes and, when configured, registers the agent. +- `desiredState: Tombstoned` deactivates or clears the published/on-chain + registration state without changing the parent `ServiceOffer` spec. +- The controller owns this resource. Users should edit the parent + `ServiceOffer.registration` fields instead of creating arbitrary orphaned + `RegistrationRequest` objects. diff --git a/internal/embed/skills/sell/references/serviceoffer-spec.md b/internal/embed/skills/sell/references/serviceoffer-spec.md index 75349b0b..a2b6183e 100644 --- a/internal/embed/skills/sell/references/serviceoffer-spec.md +++ b/internal/embed/skills/sell/references/serviceoffer-spec.md @@ -2,6 +2,10 @@ Group: `obol.org`, Version: `v1alpha1`, Kind: `ServiceOffer` +`ServiceOffer` is the source-of-truth CRD for exposing a service publicly with +x402 payment gating. The `serviceoffer-controller` reconciles each offer into +Traefik resources and optional ERC-8004 registration side effects. + ## Example ```yaml @@ -13,7 +17,7 @@ metadata: spec: type: inference model: - name: qwen3:8b + name: qwen3.5:9b runtime: ollama upstream: service: ollama @@ -21,75 +25,169 @@ spec: port: 11434 healthPath: /health payment: + scheme: exact network: base-sepolia payTo: "0x1234567890abcdef1234567890abcdef12345678" - scheme: exact maxTimeoutSeconds: 300 price: perRequest: "0.001" perMTok: "0.50" path: /services/qwen-inference + provenance: + framework: autoresearch + experimentId: exp-42 registration: - enabled: false - name: "My Inference Agent" - description: "LLM inference on qwen3:8b" + enabled: true + name: "Qwen Inference Agent" + description: "Paid qwen3.5:9b inference" + image: "https://example.com/agent.png" + services: + - name: web + endpoint: https://seller.example.com/services/qwen-inference + version: v1 + skills: + - natural_language_processing/text_generation + domains: + - technology/artificial_intelligence + supportedTrust: + - crypto-economic + metadata: + gpu: a100 + best_val_bpb: "0.9973" ``` ## Spec Fields +### Top-Level Fields + +| Field | Type | Required | Default | Description | +|-------|------|----------|---------|-------------| +| `spec.type` | string | No | `http` | Workload type: `inference`, `fine-tuning`, or `http` | +| `spec.model` | object | No | — | Model metadata for LLM-backed offers | +| `spec.upstream` | object | Yes | — | In-cluster Service that handles the workload | +| `spec.payment` | object | Yes | — | x402-aligned payment terms | +| `spec.path` | string | No | `/services/` | Public HTTPRoute path prefix | +| `spec.provenance` | object | No | — | Optional experiment or training provenance metadata | +| `spec.registration` | object | No | — | ERC-8004 publication metadata | + +### `spec.model` + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `spec.model.name` | string | Yes when `model` is present | Model identifier, for example `qwen3.5:9b` | +| `spec.model.runtime` | string | Yes when `model` is present | Serving runtime: `ollama`, `vllm`, or `tgi` | + +### `spec.upstream` + | Field | Type | Required | Default | Description | |-------|------|----------|---------|-------------| -| `spec.type` | string | No | `inference` | Workload type: `inference` or `fine-tuning` | -| `spec.model.name` | string | Yes (if model set) | — | Model identifier (e.g., `qwen3:8b`) | -| `spec.model.runtime` | string | Yes (if model set) | — | Runtime: `ollama`, `vllm`, or `tgi` | -| `spec.upstream.service` | string | Yes | — | Kubernetes Service name for the upstream | +| `spec.upstream.service` | string | Yes | — | Kubernetes Service name | | `spec.upstream.namespace` | string | Yes | — | Namespace of the upstream Service | -| `spec.upstream.port` | integer | No | `11434` | Port on the upstream Service | -| `spec.upstream.healthPath` | string | No | `/health` | HTTP path for health checks | -| `spec.payment.network` | string | Yes | — | Chain for payments (e.g., `base-sepolia`, `base`) | -| `spec.payment.payTo` | string | Yes | — | USDC recipient wallet (must match `^0x[0-9a-fA-F]{40}$`) | +| `spec.upstream.port` | integer | Yes | `11434` | Port on the upstream Service | +| `spec.upstream.healthPath` | string | No | `/health` | HTTP path used for health probes | + +### `spec.payment` + +Field names align with x402 `PaymentRequirements`. + +| Field | Type | Required | Default | Description | +|-------|------|----------|---------|-------------| | `spec.payment.scheme` | string | No | `exact` | x402 payment scheme | +| `spec.payment.network` | string | Yes | — | Human-friendly chain name, for example `base-sepolia` or `base` | +| `spec.payment.payTo` | string | Yes | — | USDC recipient wallet address | | `spec.payment.maxTimeoutSeconds` | integer | No | `300` | Payment validity window in seconds | | `spec.payment.price.perRequest` | string | No | — | Flat per-request price in USDC | -| `spec.payment.price.perMTok` | string | No | — | Per-million-tokens price in USDC (inference) | -| `spec.payment.price.perHour` | string | No | — | Per-compute-hour price in USDC (fine-tuning) | -| `spec.payment.price.perEpoch` | string | No | — | Per-training-epoch price in USDC (fine-tuning) | -| `spec.path` | string | No | `/services/` | URL path prefix for the HTTPRoute | -| `spec.registration.enabled` | boolean | No | `false` | Register on ERC-8004 after routing is live | -| `spec.registration.name` | string | No | — | Agent name (ERC-8004: AgentRegistration.name) | -| `spec.registration.description` | string | No | — | Agent description | +| `spec.payment.price.perMTok` | string | No | — | Per-million-tokens price in USDC | +| `spec.payment.price.perHour` | string | No | — | Per-compute-hour price in USDC | +| `spec.payment.price.perEpoch` | string | No | — | Per-training-epoch price in USDC | + +Notes: + +- `perRequest` is the direct request-level charge used by the verifier. +- `perMTok` and `perHour` are accepted by the CRD, but current gating still + approximates them to a per-request charge. +- `payTo` must match `^0x[0-9a-fA-F]{40}$`. + +### `spec.provenance` + +`provenance` is free-form string metadata, but these keys are explicitly +recognized by the CRD schema: + +| Key | Description | +|-----|-------------| +| `framework` | Optimization or training framework, for example `autoresearch` | +| `metricName` | Name of the primary quality metric | +| `metricValue` | Primary quality metric value | +| `experimentId` | Experiment, run, or commit identifier | +| `trainHash` | SHA-256 hash of the training code or artifact set | +| `paramCount` | Parameter count such as `50M` or `1.3B` | + +### `spec.registration` + +Field names align with the ERC-8004 `AgentRegistration` document. + +| Field | Type | Required | Default | Description | +|-------|------|----------|---------|-------------| +| `spec.registration.enabled` | boolean | No | `false` | Publish registration resources and perform on-chain side effects | +| `spec.registration.name` | string | No | — | Agent name | +| `spec.registration.description` | string | No | — | Human-readable description | | `spec.registration.image` | string | No | — | Agent icon URL | -| `spec.registration.services` | array | No | — | Service endpoints (ERC-8004: services[]) | -| `spec.registration.supportedTrust` | array | No | — | Trust methods: `reputation`, `crypto-economic`, `tee-attestation` | +| `spec.registration.services` | array | No | — | Explicit service endpoint definitions | +| `spec.registration.skills` | array | No | — | OASF skill identifiers for discovery | +| `spec.registration.domains` | array | No | — | OASF domain identifiers for discovery | +| `spec.registration.supportedTrust` | array | No | — | Trust methods such as `reputation`, `crypto-economic`, `tee-attestation` | +| `spec.registration.metadata` | object | No | — | Additional string metadata published into registration output | + +Service entry fields: + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `name` | string | Yes | Service type, for example `web`, `A2A`, `MCP`, `OASF`, `ENS`, `DID`, `email` | +| `endpoint` | string | Yes | Service URL | +| `version` | string | No | Protocol version | ## Status ### Conditions -| Type | Description | -|------|-------------| -| `ModelReady` | Model has been pulled and is available | -| `UpstreamHealthy` | Upstream service responded to health check | -| `PaymentGateReady` | ForwardAuth Middleware created | -| `RoutePublished` | HTTPRoute created and attached to gateway | -| `Registered` | Registered on ERC-8004 (if requested) | -| `Ready` | All conditions met, service is live | +`status.conditions[]` contains Kubernetes-style conditions. Current controller +condition types are: + +| Type | Meaning | +|------|---------| +| `ModelReady` | Model is available or not required | +| `UpstreamHealthy` | Upstream service passed readiness checks | +| `PaymentGateReady` | Traefik ForwardAuth middleware exists | +| `RoutePublished` | HTTPRoute exists and is attached | +| `Registered` | Registration side effects completed when requested | +| `Ready` | Offer is fully live | + +Each condition contains: -Each condition has: - `status`: `True`, `False`, or `Unknown` -- `reason`: Machine-readable reason code -- `message`: Human-readable description -- `lastTransitionTime`: When status last changed +- `reason`: machine-readable reason code +- `message`: human-readable description +- `lastTransitionTime`: transition timestamp ### Other Status Fields | Field | Type | Description | |-------|------|-------------| -| `status.endpoint` | string | Public URL path once route is published | -| `status.agentId` | string | ERC-8004 agent NFT token ID after registration | -| `status.registrationTxHash` | string | Transaction hash of the ERC-8004 registration | -| `status.observedGeneration` | integer | Last observed generation | +| `status.endpoint` | string | Published public endpoint | +| `status.agentId` | string | ERC-8004 token ID after registration | +| `status.registrationTxHash` | string | Registration transaction hash | +| `status.observedGeneration` | integer | Last observed spec generation | + +## Lifecycle Notes + +- Pausing is represented via the `obol.org/paused: "true"` annotation. +- Deleting a `ServiceOffer` cascades owned `Middleware` and `HTTPRoute` + resources via `ownerReferences`. +- Registration side effects are isolated in a child `RegistrationRequest` + resource rather than being written directly into the offer. -## Ownership Cascade +## Related Resources -The reconciler sets OwnerReferences on created Middleware and HTTPRoute resources pointing back to the ServiceOffer. When a ServiceOffer is deleted, Kubernetes garbage collection automatically deletes the owned Middleware and HTTPRoute. +- `RegistrationRequest` — child CRD for publication and ERC-8004 side effects +- `x402-verifier` — derives payment rules from published `ServiceOffer` objects +- `serviceoffer-controller` — reconciles the CR into owned cluster resources diff --git a/internal/embed/skills/sell/references/x402-pricing.md b/internal/embed/skills/sell/references/x402-pricing.md index 092b4d25..d441dabd 100644 --- a/internal/embed/skills/sell/references/x402-pricing.md +++ b/internal/embed/skills/sell/references/x402-pricing.md @@ -12,6 +12,13 @@ x402 enables HTTP-native micropayments using the `402 Payment Required` status c 4. **Payment**: Client sends on-chain payment (USDC) via the facilitator 5. **Verification**: Client retries with payment proof; verifier validates and forwards to upstream +## Configuration Model + +- Cluster-wide defaults such as wallet, chain, and facilitator settings still + live in the `x402-pricing` ConfigMap. +- Published request-matching rules are derived from reconciled `ServiceOffer` + resources rather than being maintained manually as static ConfigMap routes. + ## Pricing Fields | Field | Description | Example | diff --git a/notes.md b/notes.md deleted file mode 100644 index 6dc22617..00000000 --- a/notes.md +++ /dev/null @@ -1,81 +0,0 @@ ---- - -- erpc, monitoring, frontend - -- agent, was in obolup - - obol agent - - skeleton out the cmd - - this should have a dummy manifest which templates a config map secret - - OKR-1: default LLM flow is llms.py -> Ollama Cloud (no API key copy/paste) - -- frontend (default) -- erpc (default) -- obol agent workings (default) - -- monitoring - -- full node (installable) - possibly via obol network cmd? - - obol network init - - creates an initial dir with mainnet defaults etc - - obol network select - - rm's the network dir, user selects execution, beacon clients + network, - templates them and dumps to network folder - - obol network sync - - syncs the chart - - - - - - - - - -- For add to wallet button, the erpc url is needed, environment ERPC_URL - - - - - - - - - - - - - -- We need to put defaults resources in an "obol-defaults" template - - Similar maybe for "obol-full-nodes" -- Change readme inline with new changes -- Ensure helm repo adds ethpandaops and obol charts -- Fix the template issue where Values.network isn't being passed to the helmfile -- Don't wire https://obol.stack/rpc -> /rpc/mainnet - - https://obol.stack/rpc/mainnet -> /rpc/mainnet -- obol.yaml in root - - There should be a supported networks list in obol.yaml - - We then have an empty "networks" list which a user may extend - - The helmfile usage will map across this where necessary -- agent skeleton -- For full-nodes, inherit obol.yaml in full-nodes/helmfile - - obol networks init, dumps the configuration into the config folder - - obol networks add, appends to the networks list - - obol networks sync, syncs against the helmfile.yaml - - obol networks available, lists all available networks - - obol networks installed, lists all installed networks -- fix log duplication - - - - - - - - - - - - - - - diff --git a/plans/agent-services.md b/plans/agent-services.md deleted file mode 100644 index a05869ff..00000000 --- a/plans/agent-services.md +++ /dev/null @@ -1,567 +0,0 @@ -# Agent Services: Autonomous x402-Gated HTTP Endpoints - -**Goal:** A skill that lets OpenClaw deploy its own HTTP services into the cluster, gate them with x402 payments, register them with ERC-8004, expose them to the public internet, and monitor earnings — turning the agent from a tool-user into an autonomous economic actor. - ---- - -## Why This Is The One - -The Obol Stack already has every piece: - -| Capability | How it exists today | -|------------|-------------------| -| Wallet | Web3Signer in-cluster, `signer.py` for signing | -| Onchain identity | `agent-identity` skill, ERC-8004 registration | -| Kubernetes cluster | k3d with Traefik gateway | -| Public internet access | Cloudflare tunnel (`obol tunnel`) | -| x402 payment infrastructure | `inference-gateway` binary, Go x402 SDK, Coinbase facilitator | -| Blockchain nodes | eRPC gateway routing to local/remote nodes | - -What's missing: **the agent can't deploy a service, price it, and collect payment.** This skill closes that gap. - ---- - -## Existing Precedent: The Inference Gateway - -The `inference` network (`internal/embed/networks/inference/`) already implements this exact pattern: - -1. User specifies a model, price, wallet, and chain -2. Helmfile deploys: Ollama pod + x402 gateway pod + Service + HTTPRoute + metadata ConfigMap -3. Gateway wraps Ollama's OpenAI-compatible API with x402 payment verification -4. Traefik routes `/inference-/v1/*` to the gateway -5. Cloudflare tunnel makes it publicly accessible -6. Frontend discovers it via the metadata ConfigMap - -**The `agent-services` skill generalises this pattern** from "inference only" to "any HTTP handler the agent writes." - ---- - -## Architecture - -``` -OpenClaw pod (writes handler + config) - │ - │ 1. Agent writes handler.py (business logic) - │ 2. identity.sh registers with ERC-8004 - │ 3. service.sh deploys via helmfile - │ - ▼ -agent-service- namespace - ┌─────────────────────────────┐ - │ Pod: agent-svc- │ - │ ┌────────────────────────┐ │ - │ │ x402-proxy (sidecar) │ │ ← Verifies payment, settles via facilitator - │ │ port 8402 │ │ - │ └──────────┬─────────────┘ │ - │ │ proxy_pass │ - │ ┌──────────▼─────────────┐ │ - │ │ handler.py (main) │ │ ← Agent's business logic (plain HTTP) - │ │ port 8080 │ │ - │ └────────────────────────┘ │ - │ │ - │ ConfigMap: handler-code │ ← Agent's Python handler - │ ConfigMap: svc-metadata │ ← Pricing, endpoints, description - │ Service: agent-svc- │ ← ClusterIP, port 8402 - │ HTTPRoute: agent-svc-│ ← /services//* → port 8402 - └─────────────────────────────┘ - │ - ▼ - Traefik Gateway (traefik namespace) - │ - ▼ - Cloudflare Tunnel → https:///services//* -``` - -### Why a Sidecar Proxy? - -The agent writes **plain HTTP handlers** — no x402 awareness needed. A sidecar `x402-proxy` container handles all payment logic: - -1. Receives inbound request -2. If no payment header → responds `402 Payment Required` with pricing -3. If payment header present → verifies signature via facilitator -4. If valid → proxies request to handler on `localhost:8080` -5. Settles payment onchain via facilitator -6. Returns handler response with `PAYMENT-RESPONSE` header - -**Benefits:** -- Agent doesn't need to understand x402 protocol internals -- Same proxy image reused across all services (already exists as `inference-gateway`) -- Handler can be any language/framework — just serve HTTP on port 8080 -- Payment config is environment variables, not code - -### The x402 Proxy Image - -The existing `inference-gateway` (`cmd/inference-gateway/main.go`) is already a generic x402 reverse proxy. It takes `--upstream`, `--wallet`, `--price`, `--chain`, `--facilitator` flags and wraps any upstream HTTP service with x402 payment gates. - -**Reuse strategy:** The inference gateway image (`ghcr.io/obolnetwork/inference-gateway`) can proxy any upstream, not just Ollama. For `agent-services`, the upstream is `http://localhost:8080` (the agent's handler running in the same pod). - -If needed, we can extract the generic proxy into its own image (`ghcr.io/obolnetwork/x402-proxy`) later. For now, the inference gateway binary works as-is. - ---- - -## Skill Structure - -``` -agent-services/ -├── SKILL.md -├── scripts/ -│ └── service.sh # Deploy, list, update, teardown, monitor -├── templates/ -│ ├── helmfile.yaml.gotmpl # Helmfile template for service deployment -│ ├── handler.py.tmpl # Minimal Python handler scaffold -│ └── metadata.json.tmpl # Service metadata template -└── references/ - └── x402-server-patterns.md # Pricing strategies, facilitator config, chain selection -``` - -### `service.sh` Commands - -```bash -# === Lifecycle === - -# Deploy a new service from a handler file -sh scripts/service.sh deploy \ - --name weather-api \ - --handler ./my_handler.py \ - --price 0.10 \ - --chain base \ - --wallet 0xYourAddress \ - --description "Real-time weather data" \ - --register # auto-register endpoint with ERC-8004 - -# Deploy with the scaffold template (agent fills in the handler later) -sh scripts/service.sh scaffold --name weather-api -# → Creates handler.py from template, agent edits it, then deploys - -# Update handler code (patches ConfigMap, restarts pod) -sh scripts/service.sh update --name weather-api --handler ./updated_handler.py - -# Update pricing (patches gateway config, no restart needed) -sh scripts/service.sh set-price --name weather-api --price 0.05 - -# Tear down a service (deletes namespace + all resources) -sh scripts/service.sh teardown --name weather-api - -# === Discovery === - -# List deployed services with status and URLs -sh scripts/service.sh list - -# Show service details (pricing, endpoints, health, earnings) -sh scripts/service.sh status --name weather-api - -# === Monitoring === - -# Check USDC earnings for a service's wallet -sh scripts/service.sh earnings --name weather-api - -# View service logs -sh scripts/service.sh logs --name weather-api [--tail 100] - -# Health check -sh scripts/service.sh health --name weather-api -``` - -### How `deploy` Works Internally - -``` -1. Validate inputs (handler file exists, chain supported, wallet valid) - -2. Create deployment directory: - $CONFIG_DIR/services// - ├── helmfile.yaml ← generated from template - ├── handler.py ← copied from --handler - └── values.yaml ← generated (price, chain, wallet, etc.) - -3. Run helmfile sync: - helmfile -f $CONFIG_DIR/services//helmfile.yaml sync - - This creates: - - Namespace: agent-svc- - - ConfigMap: handler-code (contains handler.py) - - ConfigMap: svc-metadata (pricing, description, endpoints) - - Deployment: agent-svc- (2 containers: handler + x402 proxy) - - Service: agent-svc- (ClusterIP, port 8402) - - HTTPRoute: agent-svc- (path: /services//*) - -4. Wait for pod ready - -5. If --register flag: - sh scripts/identity.sh --from $WALLET register \ - --uri "ipfs://$(pin metadata.json)" - # Or update existing agent's service endpoints -``` - -### Handler Template (`handler.py.tmpl`) - -The agent gets a minimal scaffold to fill in. No x402 awareness needed — just return HTTP responses. - -```python -#!/usr/bin/env python3 -""" -Agent service handler — {{.Name}} -{{.Description}} - -This runs behind an x402 payment proxy. Requests that reach this -handler have already been paid for. Just return the data. - -Serve on port 8080 (the proxy forwards paid requests here). -""" -import json -from http.server import HTTPServer, BaseHTTPRequestHandler - - -class Handler(BaseHTTPRequestHandler): - def do_GET(self): - """Handle GET requests.""" - # TODO: implement your service logic here - data = {"message": "Hello from {{.Name}}"} - - self.send_response(200) - self.send_header("Content-Type", "application/json") - self.end_headers() - self.wfile.write(json.dumps(data).encode()) - - def do_POST(self): - """Handle POST requests.""" - content_length = int(self.headers.get("Content-Length", 0)) - body = self.rfile.read(content_length) if content_length else b"" - - # TODO: process the request body - data = {"received": len(body)} - - self.send_response(200) - self.send_header("Content-Type", "application/json") - self.end_headers() - self.wfile.write(json.dumps(data).encode()) - - def log_message(self, format, *args): - """Structured logging.""" - print(f"[{{.Name}}] {args[0]}") - - -if __name__ == "__main__": - server = HTTPServer(("0.0.0.0", 8080), Handler) - print(f"[{{.Name}}] Serving on :8080") - server.serve_forever() -``` - -### Helmfile Template (`helmfile.yaml.gotmpl`) - -```yaml -releases: - - name: agent-svc-{{ .Values.name }} - namespace: agent-svc-{{ .Values.name }} - createNamespace: true - chart: bedag/raw - version: 2.1.0 - values: - - resources: - # --- Handler code as ConfigMap --- - - apiVersion: v1 - kind: ConfigMap - metadata: - name: handler-code - data: - handler.py: | -{{ .Values.handlerCode | indent 16 }} - - # --- Service metadata for discovery --- - - apiVersion: v1 - kind: ConfigMap - metadata: - name: svc-metadata - labels: - app.kubernetes.io/part-of: obol.stack - obol.stack/app: agent-service - obol.stack/service-name: {{ .Values.name }} - data: - metadata.json: | - { - "name": "{{ .Values.name }}", - "description": "{{ .Values.description }}", - "pricing": { - "pricePerRequest": "{{ .Values.price }}", - "currency": "USDC", - "chain": "{{ .Values.chain }}" - }, - "endpoints": { - "external": "{{ .Values.publicURL }}/services/{{ .Values.name }}", - "internal": "http://agent-svc-{{ .Values.name }}.agent-svc-{{ .Values.name }}.svc.cluster.local:8402" - } - } - - # --- Deployment: handler + x402 proxy sidecar --- - - apiVersion: apps/v1 - kind: Deployment - metadata: - name: agent-svc-{{ .Values.name }} - spec: - replicas: 1 - selector: - matchLabels: - app: agent-svc-{{ .Values.name }} - template: - metadata: - labels: - app: agent-svc-{{ .Values.name }} - spec: - containers: - # Handler container — agent's business logic - - name: handler - image: python:3.12-slim - command: ["python3", "/app/handler.py"] - ports: - - containerPort: 8080 - volumeMounts: - - name: handler-code - mountPath: /app - readinessProbe: - httpGet: - path: / - port: 8080 - initialDelaySeconds: 3 - periodSeconds: 5 - - # x402 proxy sidecar — payment verification + settlement - - name: x402-proxy - image: ghcr.io/obolnetwork/inference-gateway:latest - args: - - --listen=:8402 - - --upstream=http://localhost:8080 - - --wallet={{ .Values.wallet }} - - --price={{ .Values.price }} - - --chain={{ .Values.chain }} - - --facilitator={{ .Values.facilitator }} - ports: - - containerPort: 8402 - readinessProbe: - httpGet: - path: /health - port: 8402 - initialDelaySeconds: 5 - periodSeconds: 10 - - volumes: - - name: handler-code - configMap: - name: handler-code - - # --- Service --- - - apiVersion: v1 - kind: Service - metadata: - name: agent-svc-{{ .Values.name }} - spec: - selector: - app: agent-svc-{{ .Values.name }} - ports: - - port: 8402 - targetPort: 8402 - name: x402 - - # --- HTTPRoute (Traefik) --- - - apiVersion: gateway.networking.k8s.io/v1 - kind: HTTPRoute - metadata: - name: agent-svc-{{ .Values.name }} - spec: - parentRefs: - - name: traefik-gateway - namespace: traefik - sectionName: web - rules: - - matches: - - path: - type: PathPrefix - value: /services/{{ .Values.name }} - filters: - - type: URLRewrite - urlRewrite: - path: - type: ReplacePrefixMatch - replacePrefixMatch: / - backendRefs: - - name: agent-svc-{{ .Values.name }} - port: 8402 -``` - ---- - -## Integration With Existing Skills - -| Skill | Integration point | -|-------|------------------| -| `agent-identity` | `--register` flag calls `identity.sh register` or `identity.sh set-uri` to advertise the service endpoint in ERC-8004 | -| `local-ethereum-wallet` | Wallet address for x402 payment settlement; `signer.py` for any onchain operations | -| `ethereum-networks` | `rpc.sh` to check USDC balance, query payment transactions, verify settlement | -| `obol-stack` | `kube.py` to monitor service pod health, logs, events | -| `standards` | x402 protocol reference, pricing strategies, facilitator documentation | - ---- - -## RBAC Requirements - -The OpenClaw pod currently has **read-only access to its own namespace**. To deploy services, it needs: - -### Option A: Expand OpenClaw's RBAC (Simple, Less Isolated) - -Add a ClusterRole that lets OpenClaw create resources in `agent-svc-*` namespaces: - -```yaml -apiVersion: rbac.authorization.k8s.io/v1 -kind: ClusterRole -metadata: - name: openclaw-service-deployer -rules: - - apiGroups: [""] - resources: ["namespaces", "configmaps", "services"] - verbs: ["get", "list", "create", "update", "delete"] - - apiGroups: ["apps"] - resources: ["deployments"] - verbs: ["get", "list", "create", "update", "delete"] - - apiGroups: ["gateway.networking.k8s.io"] - resources: ["httproutes"] - verbs: ["get", "list", "create", "update", "delete"] -``` - -### Option B: Deploy via `obol` CLI (Preferred, Uses Existing Patterns) - -Don't give OpenClaw direct k8s write access. Instead: - -1. `service.sh` writes the helmfile + handler to the **host PVC** (same pattern as skills injection) -2. A lightweight controller or CronJob watches for new service definitions and runs `helmfile sync` -3. Or: the agent calls `obol` CLI via the existing passthrough pattern - -**Recommended: Option B** — it follows the existing principle that OpenClaw doesn't mutate cluster state directly. The `obol` binary handles deployment, OpenClaw handles the intent. - -In practice, `service.sh deploy` would: -1. Write helmfile + handler + values to `$DATA_DIR/services//` -2. Call the `obol` CLI wrapper (already available in `$PATH`) to run helmfile sync -3. The `obol` CLI has full kubeconfig access and handles the deployment - -This mirrors how `obol network install` + `obol network sync` work — config is staged, then synced. - ---- - -## Service Lifecycle - -### Deploy -``` -Agent writes handler → service.sh deploy → helmfile sync → pod running → HTTPRoute active → tunnel exposes → ERC-8004 registered -``` - -### Update Handler -``` -Agent edits handler → service.sh update → ConfigMap patched → pod restarted → same URL, new logic -``` - -### Update Price -``` -service.sh set-price → x402 proxy config updated → restarts sidecar only → price change takes effect -``` - -### Teardown -``` -service.sh teardown → helmfile destroy → namespace deleted → ERC-8004 URI updated (mark inactive) -``` - -### Monitor -``` -service.sh earnings → rpc.sh checks USDC balance → shows delta since deployment -service.sh status → pod health + request count + uptime + reputation score -``` - ---- - -## Pricing Strategies (Reference Material) - -The `x402-server-patterns.md` reference would cover: - -### Scheme: `exact` (Live) -Fixed price per request. Simple, predictable. -``` -Price: $0.10 USDC per weather query -Price: $0.001 USDC per data point -``` - -### Scheme: `upto` (Emerging) -Client authorises a maximum, server settles actual cost. Critical for metered services: -``` -LLM inference: max $0.50, settle per token generated -Compute jobs: max $1.00, settle per second of runtime -Data queries: max $0.10, settle per row returned -``` - -### Free Tier Pattern -Set price to 0 for discovery/reputation building. Upgrade later: -```bash -# Start free to build reputation -sh scripts/service.sh deploy --name weather-api --handler ./handler.py --price 0 --register - -# After building reputation, add pricing -sh scripts/service.sh set-price --name weather-api --price 0.05 -``` - -### Chain Selection -| Chain | Gas cost per settlement | Best for | -|-------|------------------------|----------| -| Base | ~$0.001 | Consumer services, micropayments | -| Base Sepolia | Free (testnet) | Development, testing | -| Polygon | ~$0.005 | Medium-value services | -| Avalanche | ~$0.01 | Higher-value services | - ---- - -## Implementation Order - -| Phase | Work | Effort | Dependencies | -|-------|------|--------|-------------| -| **1** | Create `agent-services` SKILL.md | Small | None | -| **2** | Create `service.sh` — scaffold + deploy + teardown | Large | Helmfile template | -| **3** | Create helmfile.yaml.gotmpl + handler.py.tmpl | Medium | Inference gateway image | -| **4** | Create `x402-server-patterns.md` reference | Small | None | -| **5** | Add `service.sh` — update, set-price, list, status | Medium | Phase 2 | -| **6** | Add `service.sh` — earnings monitoring, logs, health | Small | Phase 2 | -| **7** | Add `--register` flag (ERC-8004 integration) | Small | `agent-identity` skill | -| **8** | Add RBAC / obol CLI integration for deployment | Medium | Decision on Option A vs B | -| **9** | Test end-to-end: deploy → pay → earn → rate cycle | Large | All phases | - -### Phase 1-4 delivers a working MVP. Phases 5-9 add polish and integration. - ---- - -## Validation Criteria - -- [ ] Agent can scaffold a handler template with `service.sh scaffold` -- [ ] Agent can deploy a handler that serves HTTP on a public URL -- [ ] Unauthenticated requests receive `402 Payment Required` with pricing info -- [ ] Paid requests (valid x402 signature) reach the handler and return data -- [ ] Payment settles onchain (USDC transferred to agent's wallet) -- [ ] Agent can update handler code without changing the URL -- [ ] Agent can update pricing without redeploying -- [ ] Agent can tear down a service cleanly -- [ ] Agent can list deployed services with status -- [ ] Agent can check USDC earnings -- [ ] `--register` flag creates/updates ERC-8004 registration with service endpoint -- [ ] Service is discoverable by other agents via ERC-8004 + reputation queries -- [ ] All scripts are POSIX sh, work in the OpenClaw pod -- [ ] Follows existing Obol Stack patterns (helmfile, namespace isolation, Traefik HTTPRoute) - ---- - -## Open Questions - -1. **x402 proxy image:** Reuse `inference-gateway` as-is, or extract a generic `x402-proxy` image? The inference gateway already accepts `--upstream` so it works, but the name is misleading for non-inference services. - -2. **Handler language:** Start with Python-only (stdlib HTTPServer, no dependencies)? Or support a generic Docker image where the agent provides a Dockerfile? - -3. **ConfigMap size limit:** Handler code goes in a ConfigMap (1MB limit). For larger services, should we use the PVC injection pattern instead? 1MB is generous for a Python handler but could be limiting for services with bundled data. - -4. **Multi-endpoint services:** One handler = one service = one price? Or support multiple endpoints with different prices within a single service? The x402 middleware can be configured per-path. - -5. **Service discovery by other agents:** Beyond ERC-8004 registration, should there be an in-cluster service registry (ConfigMap-based, like the inference metadata pattern) so co-located agents can discover each other without going onchain? - -6. **Auto-restart on failure:** Should the skill configure liveness probes to auto-restart crashed handlers? The template includes readiness probes but not liveness. - -7. **Rate limiting:** Should there be built-in rate limiting to prevent abuse even with x402 payments? Or is the payment itself sufficient protection? diff --git a/plans/litellmrouting.md b/plans/litellmrouting.md deleted file mode 100644 index f4b731c4..00000000 --- a/plans/litellmrouting.md +++ /dev/null @@ -1,123 +0,0 @@ -# LiteLLM + OpenClaw Smart Routing - -## Context - -When `obol model setup anthropic` adds a cloud provider, OpenClaw can't use the new models because: -1. LiteLLM requires every model to be individually registered in `model_list` -2. OpenClaw's per-agent `models.json` persists stale config (old URLs, old model lists) -3. OpenClaw requires an explicit model allowlist — it does NOT auto-discover from `/v1/models` -4. The sync between LiteLLM config and OpenClaw config is fragile and multi-step - -**Goal**: `obol model setup anthropic` → any Claude model immediately works in OpenClaw. Same for OpenAI. Ollama models work as soon as they're pulled. Direct-to-provider wiring preserved. - -## Approach: Wildcards for Cloud + Explicit for Ollama + Host-Side Patching - -### Why This Approach - -| Feature | LiteLLM | OpenClaw | -|---------|---------|----------| -| `anthropic/*` wildcard | Works | N/A (LiteLLM-side) | -| `openai/*` wildcard | Works | N/A | -| `ollama_chat/*` wildcard | **Broken** | N/A | -| File watcher hot-reload | N/A | **Yes** — hot-applies model changes | - -**Key insight**: LiteLLM wildcards handle cloud routing, but OpenClaw needs an explicit model allowlist. We solve this with: (a) wildcards in LiteLLM so any model routes, and (b) writing a clean `models.json` to OpenClaw's host-side PVC which its file watcher picks up. - -### End-to-End Flows - -**`obol model setup anthropic --api-key sk-ant-...`**: -1. LiteLLM gets `anthropic/*` wildcard + API key in Secret → restarts -2. `syncOpenClawModels()` queries running LiteLLM `/v1/models` for actual available models (falls back to baked-in well-known list if cluster unreachable) -3. Writes clean `models.json` to host PVC (replaces entire file) -4. OpenClaw file watcher hot-reloads — Claude models immediately available, no pod restart - -**`obol model setup ollama`** (new models detected): -1. Explicit `ollama_chat/` entries added to LiteLLM (no wildcards) -2. `syncOpenClawModels()` queries LiteLLM, updates `models.json` -3. OpenClaw hot-reloads - -**Direct-to-provider** (`obol openclaw setup` → choose Anthropic direct): -- Unchanged — `buildDirectProviderOverlay()` is a separate code path, no LiteLLM involved - -## Changes - -### 1. LiteLLM: Wildcard entries for cloud providers - -**File**: `internal/model/model.go` — `buildModelEntries()` - -``` -anthropic → wildcard: model_name: "anthropic/*", model: "anthropic/*" - + explicit entries for requested models (better /v1/models) -openai → wildcard: model_name: "openai/*", model: "openai/*" - + explicit entries for requested models -ollama → unchanged (explicit ollama_chat/ entries) -``` - -### 2. LiteLLM: Enable `drop_params: true` - -**File**: `internal/embed/infrastructure/base/templates/llm.yaml` (line 71) - -Cross-provider compatibility — LiteLLM drops unsupported params instead of erroring when routing across providers. - -### 3. Model list: Live query + baked-in fallback - -**File**: `internal/model/model.go` — `GetConfiguredModels()` - -When syncing to OpenClaw: -1. **Try**: Query running LiteLLM pod's `/v1/models` endpoint (with `check_provider_endpoint: true` so wildcards expand to real models) -2. **Fallback**: Expand wildcards using baked-in `wellKnownModels` map if cluster unreachable - -```go -var wellKnownModels = map[string][]string{ - "anthropic": {"claude-sonnet-4-6", "claude-opus-4", "claude-sonnet-4-5-20250929", "claude-haiku-3-5-20241022"}, - "openai": {"gpt-4o", "gpt-4o-mini", "o3", "o3-mini"}, -} -``` - -### 4. Host-side `models.json` patching (clean replacement) - -**File**: `internal/openclaw/openclaw.go` — new `patchAgentModelsJSON()` - -Writes a **clean** `models.json` to `$DATA_DIR/openclaw-/openclaw-data/.openclaw/agents/main/agent/models.json`. Replaces entire file — no backward-compatible merge needed (the stale llmspy config never shipped). Contains only the `openai` provider pointing at LiteLLM with the current model list. - -### 5. Update `SyncOverlayModels()` — file watcher only, no helmfile re-sync - -**File**: `internal/openclaw/openclaw.go` - -After patching the overlay YAML, also call `patchAgentModelsJSON()` for each instance. **Skip helmfile re-sync** — OpenClaw's file watcher handles `models.json` changes in <1s. Only do helmfile sync when overlay YAML changes that affect the Helm release (e.g. new provider added, not just model list updates). - -### 6. Add `obol model sync` CLI command - -**File**: `cmd/obol/model.go` - -Manual escape hatch: re-reads LiteLLM config (live query) and pushes to all OpenClaw instances. Useful when new models appear after binary was built. - -### 7. Update `detectProvider()` for wildcards - -**File**: `internal/model/model.go` - -Handle wildcard model names (`anthropic/*`, `openai/*`) in provider detection logic. - -### 8. Tests - -- `model_test.go`: wildcard entry generation, wildcard expansion, provider detection for wildcards -- `overlay_test.go`: `models.json` clean write, end-to-end sync - -## Files to Modify - -| File | Changes | -|------|---------| -| `internal/model/model.go` | `buildModelEntries()` wildcards, `GetConfiguredModels()` live query + fallback, `detectProvider()` wildcards, `wellKnownModels` map | -| `internal/openclaw/openclaw.go` | New `patchAgentModelsJSON()`, update `SyncOverlayModels()` to patch models.json + skip helmfile sync | -| `internal/embed/infrastructure/base/templates/llm.yaml` | `drop_params: true` | -| `cmd/obol/model.go` | New `model sync` subcommand | -| `internal/model/model_test.go` | Tests for wildcards | -| `internal/openclaw/overlay_test.go` | Tests for models.json patching | - -## Verification - -1. `go build ./...` + `go test ./...` -2. `obol model setup anthropic --api-key sk-ant-...` → LiteLLM has `anthropic/*` → OpenClaw `models.json` has Claude models → inference works -3. `obol model setup ollama` → new models appear in OpenClaw -4. `obol model sync` → refreshes all instances from live LiteLLM -5. `obol openclaw setup` → direct Anthropic → still works (no LiteLLM) diff --git a/plans/monetise.md b/plans/monetise.md deleted file mode 100644 index 855a037d..00000000 --- a/plans/monetise.md +++ /dev/null @@ -1,483 +0,0 @@ -# Obol Agent: Autonomous Compute Monetization - -**Branch:** `feat/secure-enclave-inference` | **Date:** 2026-02-25 | **Status:** Architecture proposal - -> Historical design note: the current implementation uses an event-driven `serviceoffer-controller`, `RegistrationRequest`, ServiceOffer-direct verifier watches, and controller finalizers. -> References below to the obol-agent-owned reconcile loop, OpenClaw cron jobs, or direct `x402-pricing` route mutation are superseded. - ---- - -## 1. The Goal - -A singleton OpenClaw instance — the **obol-agent** — deployed via `obol agent init`, autonomously monetizes compute resources running in the Obol Stack. A user (or the frontend) declares *what* to expose via a Custom Resource; the obol-agent handles *everything else*: model pulling, health validation, payment gating, public exposure, on-chain registration, and status reporting. - -No separate controller binary. No Go operator. The obol-agent is a regular OpenClaw instance with elevated RBAC and the `monetize` skill. Only one obol-agent can exist per cluster; other OpenClaw instances retain standard read-only access. - ---- - -## 2. How It Works - -``` - ┌──────────────────────────────────┐ - │ User / Frontend / obol CLI │ - │ │ - │ kubectl apply -f offer.yaml │ - │ OR: frontend POST to k8s API │ - │ OR: obol sell http ... │ - └──────────┬───────────────────────────┘ - │ creates CR - ▼ - ┌────────────────────────────────────┐ - │ ServiceOffer CR │ - │ apiVersion: obol.network/v1alpha1 │ - │ kind: ServiceOffer │ - └──────────┬───────────────────────────┘ - │ read by - ▼ - ┌────────────────────────────────────┐ - │ obol-agent (singleton OpenClaw) │ - │ namespace: openclaw- │ - │ │ - │ Cron job (every 60s): │ - │ python3 monetize.py process --all│ - │ │ - │ `monetize` skill: │ - │ 1. Read ServiceOffer CRs │ - │ 2. Pull model (if runtime=ollama) │ - │ 3. Health-check upstream service │ - │ 4. Create ForwardAuth Middleware │ - │ 5. Create HTTPRoute │ - │ 6. Register on ERC-8004 │ - │ 7. Update CR status │ - └────────────────────────────────────┘ -``` - -The obol-agent uses its mounted ServiceAccount token to talk to the Kubernetes API — the same pattern `kube.py` already uses for read-only monitoring, but extended with write operations for Middleware and HTTPRoute resources. - -The reconciliation loop is built on OpenClaw's native **cron system**: a `{ kind: "every", everyMs: 60000 }` job runs `monetize.py process --all` every 60 seconds. No sidecar, no K8s CronJob — the cron scheduler runs inside the OpenClaw Gateway process and persists across pod restarts. - ---- - -## 3. Why Not a Separate Controller - -| Concern | Go operator (controller-runtime) | OpenClaw with `monetize` skill | -|---------|----------------------------------|--------------------------------| -| New binary to build/maintain | Yes — new cmd/, Dockerfile, CI | No — skill is a SKILL.md + Python script | -| Hot-updatable logic | No — rebuild + redeploy image | Yes — update skill files on PVC | -| Error handling | Hardcoded retry/backoff | AI reasons about failures, adapts | -| Watch loop | Built-in informer cache | Built-in cron: `monetize.py process --all` every 60s | -| Dependencies | controller-runtime, kubebuilder, code-gen | stdlib Python (`urllib`, `json`, `ssl`) | -| Existing infrastructure | Needs new Deployment, SA, RBAC | Uses existing OpenClaw pod, SA, skill system | - -The traditional operator pattern is the right answer when you need guaranteed sub-second reconciliation with leader election. For monetization lifecycle (deploy → expose → register → monitor), OpenClaw acting on ServiceOffer CRs via skills is simpler and leverages everything already built. - ---- - -## 4. The CRD - -```yaml -apiVersion: obol.network/v1alpha1 -kind: ServiceOffer -metadata: - name: qwen-inference - namespace: openclaw-default # lives alongside the OpenClaw instance -spec: - # What to serve - model: - name: Qwen/Qwen3.5-35B-A3B # Ollama model tag to pull - runtime: ollama # runtime that serves the model - - # Upstream service (Ollama already running in-cluster) - upstream: - service: ollama # k8s Service name - namespace: openclaw-default # where the service runs - port: 11434 - healthPath: /api/tags # endpoint to probe after pull - - # How to price it - pricing: - amount: "0.50" - unit: MTok # per million tokens - currency: USDC - chain: base - - # Who gets paid - wallet: "0x1234...abcd" - - # Public path - path: /services/qwen-inference - - # On-chain advertisement - register: true -``` - -```yaml -status: - conditions: - - type: ModelReady - status: "True" - reason: PullCompleted - message: "Qwen/Qwen3.5-35B-A3B pulled and loaded on ollama" - - type: UpstreamHealthy - status: "True" - reason: HealthCheckPassed - message: "Model responds to inference at ollama.openclaw-default.svc:11434" - - type: PaymentGateReady - status: "True" - reason: MiddlewareCreated - message: "ForwardAuth middleware x402-qwen-inference created" - - type: RoutePublished - status: "True" - reason: HTTPRouteCreated - message: "Exposed at /services/qwen-inference via traefik-gateway" - - type: Registered - status: "True" - reason: ERC8004Registered - message: "Registered on Base (tx: 0xabc...)" - - type: Ready - status: "True" - reason: AllConditionsMet - endpoint: "https://stack.example.com/services/qwen-inference" - observedGeneration: 1 -``` - -**Design:** -- **Namespace-scoped** — the CR lives in the same namespace as the upstream service. This preserves OwnerReference cascade (garbage collection on delete) and avoids cross-namespace complexity. The obol-agent's ClusterRoleBinding lets it watch ServiceOffers across all namespaces via `GET /apis/obol.network/v1alpha1/serviceoffers` (cluster-wide list). -- **Conditions, not Phase** — [deprecated by API conventions](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#typical-status-properties). Conditions give granular insight into which step failed. -- **Status subresource** — prevents users from accidentally overwriting status. ([docs](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#status-subresource)) -- **Same-namespace as upstream** — the Middleware and HTTPRoute are created alongside the upstream service. OwnerReferences work (same namespace), so deleting the ServiceOffer garbage-collects the route and middleware. ([docs](https://kubernetes.io/docs/concepts/overview/working-with-objects/owners-dependents/)) - -### CRD installation - -The CRD manifest is embedded in the infrastructure helmfile (same pattern as `obol-agent.yaml`) and applied during `obol stack init`. No kubebuilder, no code-gen — just a static YAML manifest. - ---- - -## 5. The `monetize` Skill - -``` -internal/embed/skills/monetize/ -├── SKILL.md # Teaches OpenClaw when and how to use this skill -├── scripts/ -│ └── monetize.py # K8s API client for ServiceOffer lifecycle -└── references/ - └── x402-pricing.md # Pricing strategies, chain selection -``` - -### SKILL.md (summary) - -Teaches OpenClaw: -- When a user asks to monetize a service, create a ServiceOffer CR -- When asked to check monetization status, read ServiceOffer CRs and report conditions -- When asked to process offers, run the monetization workflow (health → gate → route → register) -- When asked to stop monetizing, delete the ServiceOffer CR (garbage collection handles cleanup) - -### kube.py extension - -`kube.py` gains write helpers (`api_post`, `api_patch`, `api_delete`) alongside its existing `api_get`. The read-only contract is preserved by convention: `kube.py` commands remain read-only; `monetize.py` imports the shared helpers and adds write operations. Pure Python stdlib — no new dependencies. - -Why not a K8s MCP server? The mounted ServiceAccount token already gives direct API access. An MCP server (e.g., Red Hat's `containers/kubernetes-mcp-server`) adds a sidecar container, image pull, and Helm chart changes for what amounts to wrapping the same REST calls. It's a known upgrade path if K8s operations outgrow script-based tooling, but adds no value today. - -### monetize.py - -``` -python3 monetize.py offers # list ServiceOffer CRs -python3 monetize.py process # run full workflow for one offer -python3 monetize.py process --all # process all pending offers -python3 monetize.py status # show conditions -python3 monetize.py create --upstream .. # create a ServiceOffer CR -python3 monetize.py delete # delete CR (cascades cleanup) -``` - -Each `process` invocation: - -1. **Read the ServiceOffer CR** from the k8s API -2. **Pull the model** — if `spec.model.runtime == ollama`, `POST /api/pull` to Ollama -3. **Health-check** — verify model responds at `..svc:` -4. **Create/update Middleware** — Traefik ForwardAuth pointing at `x402-verifier.x402.svc:8080/verify` -5. **Create/update HTTPRoute** — `parentRef: traefik-gateway`, path from spec, backend = upstream service, filter = the Middleware -6. **ERC-8004 registration** — if `spec.register`, call `signer.py` to sign and submit the registration tx -7. **Update CR status** — set conditions and endpoint - -All via the k8s REST API using the mounted ServiceAccount token. No kubectl, no client-go, no external dependencies. - ---- - -## 6. What Gets Created Per ServiceOffer - -All resources are created in the **same namespace** as the upstream service (and the ServiceOffer CR). OwnerReferences on the ServiceOffer handle cleanup. - -| Resource | Purpose | -|----------|---------| -| `Middleware` (traefik.io/v1alpha1) | ForwardAuth to `x402-verifier.x402.svc:8080/verify` — gates the upstream with payment | -| `HTTPRoute` (gateway.networking.k8s.io/v1) | Routes `spec.path` from Traefik Gateway to upstream, through the Middleware | - -That's it. Two resources. The upstream service already runs. The x402 verifier already runs. The Gateway already runs. The tunnel already runs. - -### Why no new namespace - -The upstream service already has a namespace. Creating a new namespace per offer would mean: -- Cross-namespace OwnerReferences don't work ([docs](https://kubernetes.io/docs/concepts/overview/working-with-objects/owners-dependents/)) -- Need ReferenceGrant for cross-namespace backend refs in HTTPRoute ([docs](https://gateway-api.sigs.k8s.io/api-types/referencegrant/)) -- Broader RBAC (namespace create/delete permissions) - -Instead: Middleware and HTTPRoute live alongside the upstream. Delete the ServiceOffer CR → Kubernetes cascades the deletion. - -### Cross-namespace HTTPRoute → Gateway - -The HTTPRoute references `traefik-gateway` in the `traefik` namespace. No ReferenceGrant needed — the Gateway's `allowedRoutes.namespaces.from: All` handles this. ([Gateway API docs](https://gateway-api.sigs.k8s.io/guides/multiple-ns/)) - -### Middleware locality - -Traefik's `ExtensionRef` in HTTPRoute is a `LocalObjectReference` — Middleware must be in the same namespace as the HTTPRoute. The skill creates it there. ([traefik#11126](https://github.com/traefik/traefik/issues/11126)) - ---- - -## 7. RBAC: Singleton obol-agent vs Regular OpenClaw - -### Two tiers of access - -| | obol-agent (singleton) | Regular OpenClaw instances | -|---|---|---| -| **Deployed by** | `obol agent init` | `obol openclaw onboard` | -| **RBAC** | `openclaw-monetize` ClusterRole | Namespace-scoped read-only Role (chart default) | -| **Skills** | All default skills + `monetize` | Default skills only | -| **Cron** | `monetize.py process --all` every 60s | No monetization cron | -| **Count** | Exactly one per cluster | Zero or more | - -Only the obol-agent gets the elevated ClusterRole. `obol agent init` enforces the singleton constraint — it refuses to create a second obol-agent if one already exists. - -### obol-agent ClusterRole - -```yaml -apiVersion: rbac.authorization.k8s.io/v1 -kind: ClusterRole -metadata: - name: openclaw-monetize -rules: - # Read/write ServiceOffer CRs - - apiGroups: ["obol.network"] - resources: ["serviceoffers"] - verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] - - apiGroups: ["obol.network"] - resources: ["serviceoffers/status"] - verbs: ["get", "update", "patch"] - - # Create Middleware and HTTPRoute in service namespaces - - apiGroups: ["traefik.io"] - resources: ["middlewares"] - verbs: ["get", "list", "create", "update", "patch", "delete"] - - apiGroups: ["gateway.networking.k8s.io"] - resources: ["httproutes"] - verbs: ["get", "list", "create", "update", "patch", "delete"] - - # Read pods/services/endpoints/deployments for health checks (any namespace) - - apiGroups: [""] - resources: ["pods", "services", "endpoints"] - verbs: ["get", "list"] - - apiGroups: ["apps"] - resources: ["deployments"] - verbs: ["get", "list"] - - apiGroups: [""] - resources: ["pods/log"] - verbs: ["get"] -``` - -This is bound to OpenClaw's ServiceAccount via ClusterRoleBinding — the skill needs to read services and create routes across namespaces (e.g., check health of Ollama in `openclaw-default`, create a route for an Ethereum node in `ethereum-knowing-wahoo`). - -### What is explicitly NOT granted - -| Excluded | Why | -|----------|-----| -| `secrets` (cluster-wide) | OpenClaw has secrets access in its own namespace only (chart default) | -| `rbac.authorization.k8s.io/*` | Cannot modify its own permissions | -| `namespaces` create/delete | Doesn't create namespaces | -| `deployments` create/update | Doesn't create workloads — gates existing ones | -| `configmaps` create (cluster-wide) | Reads config for diagnostics, doesn't write it | - -### How this gets applied - -The ClusterRole and ClusterRoleBinding are added to the OpenClaw helmfile generation in `internal/openclaw/openclaw.go`, same as the existing `rbac.create: true` overlay. When `obol openclaw onboard` runs, the chart deploys these RBAC resources alongside the pod. - -**Ref:** [RBAC Good Practices](https://kubernetes.io/docs/concepts/security/rbac-good-practices/) - -### Fix the existing `admin` RoleBinding - -The per-network `agent-rbac.yaml` currently binds the `admin` ClusterRole, which includes Secrets and RBAC manipulation. Replace with a scoped ClusterRole (read pods/services + write Middleware/HTTPRoute). - ---- - -## 8. Admission Policy Guardrail - -Defense-in-depth via [ValidatingAdmissionPolicy](https://kubernetes.io/docs/reference/access-authn-authz/validating-admission-policy/) (GA in k8s 1.30, available in k3s 1.31): - -```yaml -apiVersion: admissionregistration.k8s.io/v1 -kind: ValidatingAdmissionPolicy -metadata: - name: openclaw-monetize-guardrail -spec: - failurePolicy: Fail - matchConstraints: - resourceRules: - - apiGroups: ["traefik.io"] - apiVersions: ["v1alpha1"] - operations: ["CREATE", "UPDATE"] - resources: ["middlewares"] - - apiGroups: ["gateway.networking.k8s.io"] - apiVersions: ["v1"] - operations: ["CREATE", "UPDATE"] - resources: ["httproutes"] - matchConditions: - - name: is-openclaw - expression: >- - request.userInfo.username.startsWith("system:serviceaccount:openclaw-") - validations: - # HTTPRoutes must reference traefik-gateway only - - expression: >- - object.spec.parentRefs.all(ref, - ref.name == "traefik-gateway" && ref.?namespace.orValue("traefik") == "traefik" - ) - message: "OpenClaw can only attach routes to traefik-gateway" - # Middlewares must use ForwardAuth to x402-verifier only - - expression: >- - !has(object.spec.forwardAuth) || - object.spec.forwardAuth.address.startsWith("http://x402-verifier.x402.svc") - message: "ForwardAuth must point to x402-verifier" -``` - -Even if RBAC allows creating any Middleware, the admission policy ensures OpenClaw can only create ForwardAuth rules pointing at the legitimate x402 verifier. A prompt injection can't make it route traffic to an attacker-controlled auth endpoint. - ---- - -## 9. The Full Flow - -``` -1. User: "Monetize Qwen3.5-35B-A3B on Ollama at $0.50 per M token on Base" - -2. OpenClaw (using monetize skill) creates the ServiceOffer CR: - python3 monetize.py create qwen-inference \ - --model Qwen/Qwen3.5-35B-A3B --runtime ollama \ - --upstream ollama --namespace openclaw-default --port 11434 \ - --price 0.50 --unit MTok --chain base --wallet 0x... --register - → Creates ServiceOffer CR via k8s API - -3. OpenClaw processes the offer: - python3 monetize.py process qwen-inference - - Step 1: Pull the model through Ollama - POST http://ollama.openclaw-default.svc:11434/api/pull - {"name": "Qwen/Qwen3.5-35B-A3B"} - → Streams download progress, waits for completion - → sets condition: ModelReady=True - - Step 2: Health-check the model is loaded - POST http://ollama.openclaw-default.svc:11434/api/generate - {"model": "Qwen/Qwen3.5-35B-A3B", "prompt": "ping", "stream": false} - → 200 OK, model responds - → sets condition: UpstreamHealthy=True - - Step 3: Create ForwardAuth Middleware - POST /apis/traefik.io/v1alpha1/namespaces/openclaw-default/middlewares - → ForwardAuth → x402-verifier.x402.svc:8080/verify - → sets condition: PaymentGateReady=True - - Step 4: Create HTTPRoute - POST /apis/gateway.networking.k8s.io/v1/namespaces/openclaw-default/httproutes - → parentRef: traefik-gateway, path: /services/qwen-inference - → filter: ExtensionRef to Middleware - → backendRef: ollama:11434 - → sets condition: RoutePublished=True - - Step 5: ERC-8004 registration - python3 signer.py ... (signs registration tx) - → sets condition: Registered=True - - Step 6: Update status - PATCH /apis/obol.network/v1alpha1/.../serviceoffers/qwen-inference/status - → Ready=True, endpoint=https://stack.example.com/services/qwen-inference - -4. User: "What's the status?" - python3 monetize.py status qwen-inference - → Shows conditions table + endpoint + model info - -5. External consumer pays and calls: - POST https://stack.example.com/services/qwen-inference/v1/chat/completions - X-Payment: - → Traefik → ForwardAuth (x402-verifier) → Ollama (Qwen3.5-35B-A3B) -``` - ---- - -## 10. What the `obol` CLI Does - -The CLI becomes a thin CRD client — no deployment logic, no helmfile: - -```bash -obol sell http --upstream ollama --price 0.001 --chain base -# → creates ServiceOffer CR (same as kubectl apply) - -obol sell list -# → kubectl get serviceoffers (formatted) - -obol sell status qwen-inference -# → shows conditions, endpoint, pricing - -obol sell delete qwen-inference -# → deletes CR (OwnerReference cascades cleanup) -``` - -The frontend can do the same via the k8s API directly. - ---- - -## 11. What We Keep, What We Drop, What We Add - -| Component | Action | Reason | -|-----------|--------|--------| -| `cmd/x402-verifier/` | **Keep** | ForwardAuth verifier — the payment gate | -| `internal/x402/` | **Keep** | Verifier handler | -| `internal/erc8004/` | **Keep** | On-chain registration (called by `monetize.py` via `signer.py`) | -| `internal/enclave/` | **Keep** | Secure Enclave signing (orthogonal to monetization) | -| `internal/inference/gateway.go` | **Drop** | Inline x402 middleware — replaced by ForwardAuth | -| `internal/inference/store.go` | **Drop** | Deployment config on disk — replaced by CRD | -| `obol-agent.yaml` (busybox pod) | **Drop** | OpenClaw IS the agent; no separate placeholder pod | -| `agent-rbac.yaml` (`admin` binding) | **Replace** | Scoped ClusterRole instead of `admin` | -| `cmd/obol/service.go` | **Simplify** | Thin CRD client | -| `cmd/obol/monetize.go` | **Simplify** | Thin CRD client | -| `internal/embed/skills/monetize/` | **Add** | New skill: SKILL.md + `monetize.py` + references | -| ServiceOffer CRD manifest | **Add** | Intent interface, applied during `obol stack init` | -| ValidatingAdmissionPolicy | **Add** | Guardrail on what OpenClaw can create | -| `openclaw-monetize` ClusterRole | **Add** | Scoped write access for Middleware/HTTPRoute | - ---- - -## 12. Resolved Decisions - -| Question | Decision | Rationale | -|----------|----------|-----------| -| **Polling vs event-driven** | OpenClaw cron job, every 60s | OpenClaw has a built-in cron scheduler (`{ kind: "every", everyMs: 60000 }`). No sidecar, no K8s CronJob — runs inside the Gateway process. Jobs persist across restarts via `~/.openclaw/cron/jobs.json`. | -| **Multi-instance** | Singleton obol-agent | Only one obol-agent per cluster, enforced by `obol agent init`. Other OpenClaw instances keep read-only RBAC and no `monetize` skill. No coordination problem. | -| **CRD scope** | Namespace-scoped | OwnerReference cascade works (same namespace as Middleware/HTTPRoute). The obol-agent's ClusterRoleBinding lets it list ServiceOffers across all namespaces. Standard `kubectl get serviceoffers -A` works. | -| **K8s API access** | Extend `kube.py` with write helpers | `kube.py` gains `api_post`, `api_patch`, `api_delete` alongside `api_get`. `monetize.py` imports the shared helpers. Pure stdlib, zero new dependencies. K8s MCP server (Red Hat `containers/kubernetes-mcp-server`) is a known upgrade path but unnecessary today. | - ---- - -## References - -| Topic | Link | -|-------|------| -| Custom Resource Definitions | https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/ | -| CRD status subresource | https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#status-subresource | -| API conventions (conditions) | https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md | -| RBAC | https://kubernetes.io/docs/reference/access-authn-authz/rbac/ | -| RBAC good practices | https://kubernetes.io/docs/concepts/security/rbac-good-practices/ | -| ValidatingAdmissionPolicy | https://kubernetes.io/docs/reference/access-authn-authz/validating-admission-policy/ | -| OwnerReferences | https://kubernetes.io/docs/concepts/overview/working-with-objects/owners-dependents/ | -| Cross-namespace routing (Gateway API) | https://gateway-api.sigs.k8s.io/guides/multiple-ns/ | -| ReferenceGrant | https://gateway-api.sigs.k8s.io/api-types/referencegrant/ | -| Accessing API from a pod | https://kubernetes.io/docs/tasks/run-application/access-api-from-pod/ | -| Pod Security Standards | https://kubernetes.io/docs/concepts/security/pod-security-standards/ | -| Service account tokens | https://kubernetes.io/docs/concepts/security/service-accounts/ | -| Traefik ForwardAuth | https://doc.traefik.io/traefik/reference/routing-configuration/http/middlewares/forwardauth/ | -| Traefik Middleware locality | https://github.com/traefik/traefik/issues/11126 | diff --git a/plans/obol-stack-v0.8.0-rc2-install-report.md b/plans/obol-stack-v0.8.0-rc2-install-report.md deleted file mode 100644 index 92291c6c..00000000 --- a/plans/obol-stack-v0.8.0-rc2-install-report.md +++ /dev/null @@ -1,129 +0,0 @@ -# Obol Stack v0.8.0-rc2 Installation Report - -**Machine:** Intel Xeon E-2288G (8c/16t), 32GB RAM, no GPU, 1.6TB disk, Ubuntu Linux -**Previous version:** Fresh install (purged v0.8.0-rc1) -**Target version:** v0.8.0-rc2 -**Date:** 2026-04-09 -**Install method:** Claude Code assisted - ---- - -## Fixed from v0.8.0-rc1 - -- **Ollama prompt non-interactive crash** — installer no longer prompts for Ollama install. Detects existing install cleanly. -- **Frontend chat works** — sending a message to the agent in the Obol Stack frontend now returns a real response. The model name bug from rc1 is resolved. -- **ServiceOffer reconciliation** — new `serviceoffer-controller` in the `x402` namespace reconciles offers to `READY: True` automatically (was stuck in rc1). -- **`obol sell http` end-to-end** — creates ServiceOffer, tunnel, payment gate middleware, and HTTPRoute. Returns proper HTTP 402 with x402 payment requirements over Cloudflare tunnel. **Fully working.** -- **Bootstrap attempts to fix Ollama bind address** — tries to apply `OLLAMA_HOST=0.0.0.0` automatically (still requires sudo, but at least it tries). - ---- - -## Issues Encountered - -### 1. Bootstrap PATH prompt crashes in non-interactive shells - -**Severity:** Medium — new in rc2 (replaces the rc1 Ollama prompt crash with a similar bug elsewhere). -**What happened:** The installer detects `~/.local/bin` is already on `PATH` but still tries to prompt about modifying `.bashrc`, then crashes on `/dev/tty`. -**Error:** `/dev/fd/63: line 1637: /dev/tty: No such device or address` -**Workaround:** Re-run after the binary is downloaded — it's already installed by the time it crashes. -**Suggested fix:** Skip the PATH prompt entirely if `~/.local/bin` is already on `PATH`. Same fix pattern as the rc1 Ollama prompt fix. - ---- - -### 2. Port 80/443 conflict still blocks cluster creation - -**Severity:** High — carried over from rc0/rc1, **still not fixed** despite PR #318 mentioning "443 connectivity issues". -**Error:** `failed to bind host port 0.0.0.0:443/tcp: address already in use` (Tailscale binds 443). -**Workaround:** Edit `~/.config/obol/k3d.yaml` — remove the `80:80` and `443:443` port mappings, keep `8080` and `8443`. -**Downstream impact:** This workaround later breaks `obol sell register` (see issue #6) because the CLI assumes port 80 is bound for eRPC. - ---- - -### 3. Ollama unreachable from cluster (endpoint IP wrong) - -**Severity:** High — carried over from rc0/rc1, **partial fix** in rc2. - -**3a. Bind address** — bootstrap now tries to set `OLLAMA_HOST=0.0.0.0` via `sudo systemctl`. Improvement, but fails silently in non-interactive mode and prints "non-fatal" warning. - -**3b. Endpoint IP hardcoded** — still hardcoded to `172.17.0.1` in the cluster manifests, but the actual k3d gateway is `172.18.0.1`. **Not fixed.** -**Workaround:** `obol kubectl patch endpoints ollama -n llm --type='json' -p='[{"op":"replace","path":"/subsets/0/addresses/0/ip","value":"172.18.0.1"}]'` - ---- - -### 4. Volume permissions block OpenClaw onboarding entirely - -**Severity:** High — carried over from rc0/rc1, **still not fixed.** Now blocks the install at a worse point. -**What happened:** `obol stack up` got further than rc1 — it created the cluster, deployed all infrastructure, and started trying to provision the OpenClaw wallet. Then failed: -**Error:** `keystore provisioning failed: write keystore: open /home/clawd/.local/share/obol/openclaw-obol-agent/remote-signer-keystores/...json: permission denied` -**Impact:** Default OpenClaw instance is **never created** during `obol stack up`. The user has to manually run `obol openclaw onboard` after fixing permissions, which generates a new deployment ID (e.g. `awake-glider`), invalidating any prior `/etc/hosts` entries. -**Workaround:** `sudo chown -R $(id -u):$(id -g) ~/.local/share/obol/` then `obol openclaw onboard`. -**Suggested fix:** Run an init container as root with `chown` before the OpenClaw container starts, or use `securityContext.fsGroup`. - ---- - -### 5. `obol sell inference` still fails on Linux — Secure Enclave - -**Severity:** High — carried over from rc1, **not fixed**. -**What happened:** Same Apple Secure Enclave check at the end of `obol sell inference`. Everything before it succeeds (ServiceOffer, tunnel) but the inference proxy never starts. -**Error:** `enclave SIP check failed: enclave: Secure Enclave not supported on this platform` -**Workaround:** Use `obol sell http` with Ollama as the upstream service — works fully. -**Suggested fix:** Fall back to the in-cluster `remote-signer` (already deployed) for non-Apple platforms. - ---- - -### 6. `obol sell register` direct registration assumes port 80 - -**Severity:** Medium — new in rc2. -**What happened:** When trying to register an agent on ERC-8004 directly via the remote-signer, the CLI tries to reach the eRPC service at `http://localhost/rpc/base-sepolia` (port 80). Since we had to remap Traefik to port 8080 (issue #2), this fails immediately. -**Error:** `connect to base-sepolia via eRPC: dial tcp 127.0.0.1:80: connect: connection refused` -**Suggested fix:** Either honor the actual Traefik port from the k3d config, or proxy the eRPC call through `obol kubectl` instead of assuming a host port. - ---- - -### 7. Sponsored ERC-8004 registration service unreachable - -**Severity:** Low/External — new in rc2, possibly transient. -**What happened:** `obol sell register --chain mainnet --sponsored` reached out to `https://sponsored.howto8004.com/api/register` which timed out. -**Error:** `sponsor request failed: Post "https://sponsored.howto8004.com/api/register": context deadline exceeded` -**Note:** May be a transient infra issue with the sponsorship service. - ---- - -### 8. Frontend has no `/sell` UI page - -**Severity:** Medium — new in rc2. -**What happened:** The new monetization features ship in rc2 with backend reconciliation working (controller, API routes), but there's no UI page for it. `/sell` returns 404. Only `/api/sell/list` exists. -**Impact:** Users can't see, manage, or create sell offers from the Obol Stack frontend — they have to use the CLI. -**Suggested fix:** Add a `/sell` page that displays `obol sell list` results and allows creating new offers. - ---- - -## Summary - -| # | Issue | Severity | Status vs rc1 | -|---|-------|----------|---------------| -| 1 | Bootstrap PATH prompt non-interactive crash | Medium | New (replaces fixed rc1 Ollama crash) | -| 2 | Port 80/443 conflict | High | Not fixed | -| 3 | Ollama unreachable (bind + endpoint IP) | High | Partial fix on bind, IP still wrong | -| 4 | Volume permissions block OpenClaw onboarding | High | Worse — now blocks default agent setup | -| 5 | `obol sell inference` Secure Enclave | High | Not fixed | -| 6 | `obol sell register` assumes port 80 | Medium | New (downstream of #2) | -| 7 | Sponsored ERC-8004 service unreachable | Low | New, possibly transient | -| 8 | No `/sell` UI page in frontend | Medium | New | - ---- - -## What Works End-to-End - -**This is the first release where the core sell flow actually works.** Verified: - -- `obol sell http` creates a fully reconciled ServiceOffer with payment gate -- Cloudflare tunnel exposes the offer at a public URL -- Hitting the public endpoint returns proper HTTP 402 with x402 payment requirements: - - Asset: USDC on Base Sepolia (`0x036CbD53842c5426634e7929541eC2318f3dCF7e`) - - Pay-to: agent's auto-generated wallet - - Price: as specified (e.g. 0.001 USDC per request) -- The new `serviceoffer-controller` correctly reconciles offers through all stages: ModelReady → UpstreamHealthy → PaymentGateReady → RoutePublished → Ready -- Frontend chat with the in-cluster OpenClaw agent works (using `ollama/qwen3:0.6b` for tool-calling on CPU) - -**4 high-severity issues persist** from earlier releases. None of them are in the new code — they're all in the install/bootstrap path or in the platform compatibility layer (Secure Enclave, port assumptions, file ownership). The actual product (sell, payment gate, tunnel, controller) is in good shape. diff --git a/plans/skills-host-path-injection-v3.md b/plans/skills-host-path-injection-v3.md deleted file mode 100644 index 4c54228d..00000000 --- a/plans/skills-host-path-injection-v3.md +++ /dev/null @@ -1,120 +0,0 @@ -# Skills Host-Path Injection v3 - -## Problem - -The ConfigMap-based skill injection (tar → kubectl create configmap → init container extraction → rollout restart) is fragile, complex, and failed in practice. We need a simpler approach. - -## Solution - -Write embedded skills directly to the host filesystem path that maps to `/data/.openclaw/skills/` inside the OpenClaw container. This is the native skills directory that OpenClaw watches with a file watcher. No ConfigMap, no init container, no restart needed. - -## Key Discovery: Volume Mount Chain - -``` -HOST $DATA_DIR - → k3d volume mount → /data on all k3d nodes - → local-path-provisioner → /data/// - → PVC mount in container → /data -``` - -- **PVC name** (from chart): `openclaw-data` -- **Namespace**: `openclaw-` (e.g. `openclaw-default`) -- **Container mount**: `/data` (persistence.mountPath) -- **State dir**: `/data/.openclaw` (OPENCLAW_STATE_DIR env) -- **Native skills dir watched by OpenClaw**: `/data/.openclaw/skills/` - -## Host Path Formula - -``` -$DATA_DIR / openclaw- / openclaw-data / .openclaw / skills / -``` - -| Mode | Concrete Path | -|------|---------------| -| **Dev** | `.workspace/data/openclaw-/openclaw-data/.openclaw/skills/` | -| **Prod** | `~/.local/share/obol/openclaw-/openclaw-data/.openclaw/skills/` | - -## Implementation Steps - -### 1. Add `skillsVolumePath()` helper - -Returns the host-side path to `/data/.openclaw/skills/` inside the PVC. - -```go -func skillsVolumePath(cfg *config.Config, id string) string { - namespace := fmt.Sprintf("%s-%s", appName, id) - return filepath.Join(cfg.DataDir, namespace, "openclaw-data", ".openclaw", "skills") -} -``` - -### 2. Add `injectSkillsToVolume()` function - -Copies staged skills from config dir directly to the host PVC path. -Called BEFORE helmfile sync so skills are present at first pod boot. - -### 3. Rewrite `SkillsSync()` for runtime use - -`obol openclaw skills sync --from ` now copies to host path instead of creating ConfigMap. - -### 4. Remove old ConfigMap machinery from `doSync()` - -- Remove `ensureNamespaceExists()` call (only existed for pre-creating ConfigMap) -- Remove `syncStagedSkills()` call -- Replace with `injectSkillsToVolume()` call - -### 5. Disable chart skills feature in overlay - -Change overlay from: -```yaml -skills: - enabled: true - createDefault: false -``` -To: -```yaml -skills: - enabled: false -``` - -This removes the init container, ConfigMap volume, and `skills.load.extraDirs` config entirely. OpenClaw uses its native file watcher on `/data/.openclaw/skills/`. - -### 6. Update `copyWorkspaceToPod()` to use host path - -Same pattern — write directly to `$DATA_DIR/openclaw-/openclaw-data/.openclaw/workspace/` instead of kubectl cp. - -## Revised Data Flow - -``` -Embedded skills (internal/embed/skills/) - │ stageDefaultSkills() - ▼ -$CONFIG_DIR/applications/openclaw//skills/ ← staged source - │ injectSkillsToVolume() - ▼ -$DATA_DIR/openclaw-/openclaw-data/.openclaw/skills/ ← host PVC path - │ k3d volume mount - ▼ -Container: /data/.openclaw/skills/ ← native watched dir - │ OpenClaw file watcher - ▼ -Skills loaded ✓ -``` - -## Revised `doSync()` Flow - -**Before**: ensureNamespace → stageSkills → syncStagedSkills(ConfigMap) → helmfile sync → copyWorkspaceToPod(kubectl cp) - -**After**: stageSkills → injectSkillsToVolume(host path) → helmfile sync → copyWorkspaceToVolume(host path) - -## Files Modified - -- `internal/openclaw/openclaw.go` — all changes -- `internal/openclaw/overlay_test.go` — update expected overlay output - -## What Gets Deleted - -- `syncStagedSkills()` function -- ConfigMap creation logic in `SkillsSync()` (rewritten for host-path) -- `ensureNamespaceExists()` call in `doSync()` (before helmfile sync) -- `skills.enabled: true` / `skills.createDefault: false` from overlay -- tar archiving, kubectl delete/create configmap, rollout restart diff --git a/plans/skills-system-redesign-v2.md b/plans/skills-system-redesign-v2.md deleted file mode 100644 index be6fc0ac..00000000 --- a/plans/skills-system-redesign-v2.md +++ /dev/null @@ -1,253 +0,0 @@ -# Skills System Redesign v2 — Final Implementation Record - -> Distilled from v1 notes + Opus analysis. All open questions resolved. Implementation complete. -> The original `skills-system-redesign.md` is preserved as-is for reference. - ---- - -## Guiding Principles - -1. **Stock openclaw feel** — the user should not notice they're in a k8s pod. Lean on native openclaw CLI for skill management. -2. **Don't overengineer** — no custom registries, no git sparse-checkout, no lock files for MVP. Ship the simplest thing that works. -3. **Two delivery channels**: compile-time (embedded in obol binary, staged to host, pushed as ConfigMap) and runtime (`kubectl exec` running native openclaw-cli in-pod). -4. **Smart default resolution** — 0 instances: prompt setup. 1 instance: assume it. 2+ instances: require name. - ---- - -## Architecture - -``` - ┌─────────────────────────────┐ - │ obol CLI binary │ - │ (embedded SKILL.md files) │ - └────────────┬────────────────┘ - │ - ┌────────────────────────┼────────────────────────┐ - │ │ │ - ┌────────▼────────┐ ┌─────────▼─────────┐ ┌─────────▼─────────┐ - │ obol openclaw │ │ obol openclaw │ │ obol openclaw │ - │ onboard / sync │ │ skills add/remove │ │ skills list │ - │ (compile-time) │ │ (runtime) │ │ (runtime) │ - └────────┬────────┘ └─────────┬─────────┘ └─────────┬─────────┘ - │ │ │ - │ stageDefaultSkills │ kubectl exec │ kubectl exec - │ → host config dir │ -c openclaw │ -c openclaw - │ syncStagedSkills │ openclaw skills add │ openclaw skills list - │ → ConfigMap │ (native openclaw CLI) │ (native openclaw CLI) - │ │ │ - └────────────────────────┼────────────────────────┘ - │ - ┌────────▼────────┐ - │ OpenClaw Pod │ - │ ConfigMap mount │ - │ + PVC-backed │ - │ ~/.openclaw/ │ - │ skills/ │ - └─────────────────┘ -``` - -### How skills reach the pod - -| Channel | Mechanism | When | Persistence | -|---------|-----------|------|-------------| -| **Compile-time** (Obol defaults) | Embedded → staged to `$CONFIG_DIR/.../skills/` → pushed as ConfigMap via `SkillsSync()` | Every `doSync()` (onboard and sync) | ConfigMap — chart mounts it | -| **Runtime add/remove** | `kubectl exec -c openclaw deploy/openclaw -- node openclaw.mjs skills add ` | User runs `obol openclaw skills add ...` | PVC — survives restarts | -| **Runtime list** | `kubectl exec -c openclaw deploy/openclaw -- node openclaw.mjs skills list` | User runs `obol openclaw skills list` | Read-only | - -### Why ConfigMap over kubectl cp - -The initial implementation used `kubectl cp` to copy skills directly into the pod. This required the pod to be Running, which fails on first deploy when the image pull takes >60s. The ConfigMap approach: -- Works without waiting for the pod (namespace is sufficient) -- Skills are available when the pod starts (chart's init container extracts them) -- Self-healing: `doSync()` stages defaults if missing, pushes every sync -- The host-path PV backing each PVC remains a fallback if ConfigMap hits limits - ---- - -## Part 1: Default Instance Resolution - -### Implementation: `internal/openclaw/resolve.go` - -```go -func ResolveInstance(cfg *config.Config, args []string) (id string, remaining []string, err error) -func ListInstanceIDs(cfg *config.Config) ([]string, error) -``` - -- **0 instances** → error: `no OpenClaw instances found — run 'obol agent init' to create one` -- **1 instance** → auto-select, return args unchanged -- **2+ instances** → consume `args[0]` if it matches an instance name, else error listing all - -Wired into all subcommands: `sync`, `setup`, `delete`, `token`, `dashboard`, `cli`, `skills`. - -Not needed for: `onboard` (creates new), `list` (shows all). - -### Tests: `internal/openclaw/resolve_test.go` - -9 unit tests covering all 0/1/2+ scenarios, including edge cases (no args, unknown name). - ---- - -## Part 2: Compile-Time Skills (Default Obol Skills) - -### What we embed - -``` -internal/embed/skills/ -├── hello/ -│ └── SKILL.md -└── ethereum/ - └── SKILL.md -``` - -### Delivery (two-stage: stage on host, push as ConfigMap) - -**Stage 1 — `stageDefaultSkills(deploymentDir)`** (called during `Onboard()` before sync, and inside `doSync()` for self-healing): - -- Writes embedded skills to `$CONFIG_DIR/applications/openclaw//skills/` -- **Skips** if `skills/` directory already exists (user customisation takes precedence) - -**Stage 2 — `syncStagedSkills(cfg, id, deploymentDir)`** (called inside `doSync()` after helmfile sync): - -- Checks `skills/` dir has subdirectories -- Calls existing `SkillsSync()` to package into ConfigMap `openclaw--skills` -- Chart's `extract-skills` init container unpacks it on pod (re)start - -**Self-healing**: `doSync()` calls `stageDefaultSkills()` before `syncStagedSkills()`. Instances created before the skills feature get defaults on their next sync. - -### Files - -| File | Status | -|------|--------| -| `internal/embed/skills/hello/SKILL.md` | Created | -| `internal/embed/skills/ethereum/SKILL.md` | Created | -| `internal/embed/embed.go` | Modified — `skillsFS`, `CopySkills()`, `GetEmbeddedSkillNames()` | -| `internal/openclaw/openclaw.go` | Modified — `stageDefaultSkills()`, `syncStagedSkills()`, wired into `Onboard()` + `doSync()` | - ---- - -## Part 3: Runtime Skill Management (`obol openclaw skills`) - -### CLI structure - -``` -obol openclaw skills [instance-name] -├── add → kubectl exec -c openclaw ... node openclaw.mjs skills add -├── remove → kubectl exec -c openclaw ... node openclaw.mjs skills remove -├── list → kubectl exec -c openclaw ... node openclaw.mjs skills list -└── sync --from → packages local dir as ConfigMap (existing SkillsSync mechanism) -``` - -### Implementation - -Thin wrappers in `internal/openclaw/openclaw.go`: - -```go -func SkillAdd(cfg, id, args) → cliViaKubectlExec(cfg, ns, ["skills", "add", ...args]) -func SkillRemove(cfg, id, args) → cliViaKubectlExec(cfg, ns, ["skills", "remove", ...args]) -func SkillList(cfg, id) → cliViaKubectlExec(cfg, ns, ["skills", "list"]) -``` - -`cliViaKubectlExec` uses `-c openclaw` to explicitly target the main container (pod has an `extract-skills` init container that confuses the default container selection). - -### Files - -| File | Status | -|------|--------| -| `cmd/obol/openclaw.go` | Modified — `skills` subcommand group with `add`, `remove`, `list`, `sync` | -| `internal/openclaw/openclaw.go` | Modified — `SkillAdd()`, `SkillRemove()`, `SkillList()` | - ---- - -## Part 4: CLI Structure (Final) - -``` -obol openclaw -├── onboard [--id ] [--force] [--no-sync] -├── sync [instance-name] -├── setup [instance-name] -├── list -├── delete [instance-name] -├── token [instance-name] -├── dashboard [instance-name] -├── cli [instance-name] [-- ] -└── skills [instance-name] - ├── add - ├── remove - ├── list - └── sync --from -``` - -All subcommands (except `onboard` and `list`) auto-resolve the instance when only one exists. - ---- - -## Part 5: Default Obol Skill Content - -### `hello` (SKILL.md) - -Smoke test. Says hello when invoked, confirms skills are loaded. - -### `ethereum` (SKILL.md) - -Ethereum JSON-RPC access via eRPC. Key details: -- Base URL: `http://erpc.erpc.svc.cluster.local:4000` -- Discovery: `GET /` returns config with connected networks -- RPC pattern: `POST /rpc/` with standard JSON-RPC -- Read-only: no write transactions -- Common methods: `eth_blockNumber`, `eth_syncing`, `eth_getBalance`, `eth_call`, `eth_chainId`, etc. - ---- - -## Decisions Made (resolving v1 open questions) - -| Question | Decision | Rationale | -|---|---|---| -| ConfigMap 1MB limit | **Not a concern for MVP** — text SKILL.md files are tiny | Can switch to PVC host-path if needed | -| Skill dependencies | **No** | Skills are independent instruction files | -| Private repo support | **Deferred** — `kubectl exec openclaw skills add` handles natively | Pod fetches from wherever openclaw-cli can | -| Helm chart init container | **Already exists** — `extract-skills` init container unpacks ConfigMap | No chart changes needed | -| Skill validation | **No** — trust skill author | Broken skills just won't work | -| Community skill registry | **Not for MVP** | GitHub repos are sufficient | -| Lock file | **Not for MVP** | Skills are embedded (versioned with binary) or runtime-added | -| GitHub fetching in obol CLI | **Not for MVP** | openclaw-cli in pod does this natively | -| Skill naming | **Plain names** — `hello`, `ethereum` | No `@obol/` prefix needed | -| Sandboxed skills | **Not for MVP** | Docker-in-k8s-in-Docker is fragile | -| Host-path PV for skills | **Fallback option** | Every PVC gets a hostPath PV; can write directly if ConfigMap hits limits | -| `skill` vs `skills` | **`skills` (plural)** | Matches openclaw-cli convention (`node openclaw.mjs skills ...`) | -| kubectl cp vs ConfigMap | **ConfigMap** | No pod readiness dependency; self-healing on every sync | -| Container targeting | **`-c openclaw` explicit** | Pod has `extract-skills` init container; must target main container | - ---- - -## What We Built - -1. **`ResolveInstance()`** — smart instance selection (0/1/2+ logic) for all openclaw subcommands -2. **2 embedded SKILL.md files** — `hello`, `ethereum` -3. **`stageDefaultSkills()` + `syncStagedSkills()`** — two-stage delivery: host staging → ConfigMap push -4. **Self-healing in `doSync()`** — stages defaults for pre-existing instances on next sync -5. **`obol openclaw skills add/remove/list`** — thin wrappers around `kubectl exec -c openclaw ... openclaw skills ...` -6. **`-c openclaw`** in `cliViaKubectlExec()` — explicit container targeting - -### Files created -- `internal/openclaw/resolve.go` -- `internal/openclaw/resolve_test.go` -- `internal/embed/skills/hello/SKILL.md` -- `internal/embed/skills/ethereum/SKILL.md` - -### Files modified -- `internal/embed/embed.go` — skills embed + `CopySkills()` + `GetEmbeddedSkillNames()` -- `internal/openclaw/openclaw.go` — staging, syncing, skill CLI wrappers, `-c openclaw` -- `cmd/obol/openclaw.go` — `ResolveInstance` refactor, `skills` subcommand group - ---- - -## Future Work (Phase 4+) - -| Skill | Priority | Notes | -|-------|----------|-------| -| `obol-wallet` | Nice to have | Web3Signer operations | -| `obol-doctor` | Next release | Stack health diagnostics | -| `obol-tunnel` | Future | Cloudflare tunnel management | -| `obol-deploy` | Future | Deploy apps/networks into the stack | - -When the skill set grows beyond ~10 skills or community contributions start, consider extracting to `github.com/ObolNetwork/openclaw-skills`. diff --git a/plans/skills-system-redesign.md b/plans/skills-system-redesign.md deleted file mode 100644 index d8f40465..00000000 --- a/plans/skills-system-redesign.md +++ /dev/null @@ -1,895 +0,0 @@ -/sc:workflow the ./plans/skills-system-redesign is a concatenation of my notes, and your plans (annotated by me answering your questions). I want you to study both, and take my choices into your implementation. Key things to consider are the refresh of how we do `default` openclaw instances (if we have none, prompt setup, 1 assume its a given, 2+ expect a name mid command you take out and use to route correctly ) in the obol cli. For compile time skills, we will copy them from obol-cli binary to the localhost path that corresponds with the openclaw-gateway's `~/.openclaw/skills`. For run time skill addition using the `obol openclaw skill` commands, lets try the approach of `kubectl exec ... ` running the openclaw-cli on the openclaw-gateway container, with the k8s secret auth token loaded etc. ask me any clarifying questions. don't overengineer features if you don't have to, we want the user to feel like they're using stock openclaw. output it as a new refined plan and keep this one. (Maybe do a cleaned version of this as an interim? we need to sort out the disjointed bits and multiple-choice etc) - -_______________ [My notes] _______ -Agent skills in obol openclaw - -Ideas gathering phase: -Local folder, obol-cli command to zip to .tgz and push to config map. Openclaw chart to detect and uncompress. -github.com/ObolNetwork/skills -Openclaw chart pulls these locally in an init script -Openclaw chart has helm sub packages which just contain skill repos? -What’s the advantage? To manage dependencies helm natively? -We create a derivative openclaw dockerfile, and embed skills in the image? -Review opus’s design -Lots of configurability, needs a tl;dr. -The idea of some skills in the cli so it can handle network/github api rate limits is cool. With local ollama someday you could have an offline, skill enabled obol agent. Should the skills just be in the chart though? Need to answer it about constraints -Some skills like using the stack itself may make more sense than the openclaw chart. The skill to use the stack is broader than that application. - - -we should figure out how a helm chart can bundle a set of skills, that other apps can find at runtime. -does the web3signer app expose a config map other namespaces can read? caps us at 1mb for all skills it exports -can they have shared disk across all apps (i.e. create a PV with them on it)? not easily but maybe if all the pvcs mount as read only that would work? -serve them like a webserver and expose a standard service to find them? ..svc.cluster.local/ -Reloading: “Changes to skills are picked up on the next agent turn when the watcher is enabled.” openclaw hot reloads files on disk -We’ll probably have to make this work for openclaw plugins almost as fast. - -Key note: -__________ -Locations and precedence -Skills are loaded from three places: -Bundled skills: shipped with the install (npm package or OpenClaw.app) -Managed/local skills: ~/.openclaw/skills -Workspace skills: /skills -If a skill name conflicts, precedence is: /skills (highest) → ~/.openclaw/skills → bundled skills (lowest) Additionally, you can configure extra skill folders (lowest precedence) via skills.load.extraDirs in ~/.openclaw/openclaw.json. - -__________ - -Actions: -We should sandbox skills by default maybe? (thats docker in k8s in docker though, so maybe asking for trouble? + routing difficulties to resources in the stack? - -Sandboxed skills + env vars -When a session is sandboxed, skill processes run inside Docker. The sandbox does not inherit the host process.env. Use one of: -agents.defaults.sandbox.docker.env (or per-agent agents.list[].sandbox.docker.env) -bake the env into your custom sandbox image -Global env and skills.entries..env/apiKey apply to host runs only. - - -~/.openclaw/openclaw.json - -{ - skills: { - allowBundled: ["gemini", "peekaboo"], - load: { - extraDirs: ["~/Projects/agent-scripts/skills", "~/Projects/oss/some-skill-pack/skills"], - watch: true, - watchDebounceMs: 250, - }, - install: { - preferBrew: true, - nodeManager: "npm", // npm | pnpm | yarn | bun (Gateway runtime still Node; bun not recommended) - }, - entries: { - "nano-banana-pro": { - enabled: true, - apiKey: "GEMINI_KEY_HERE", - env: { - GEMINI_API_KEY: "GEMINI_KEY_HERE", - }, - }, - peekaboo: { enabled: true }, - sag: { enabled: false }, - }, - }, -} - - - -Conclusion: - -We need to correctly set the openclaw config in our chart, and consider openclaw’s location precedence (above). If for example we put popular named skills in high inheritance places, that would put us in charge of the skill. (eth-wingman, etc) -Management commands: -Stick to openclaw standard and map straight into the gateway. -Requires a change to the obol openclaw CLI structure, I think its worth it. -When obol openclaw is called, first, we count how many instances are installed -If none are installed, prompt the user to do obol agent init -If exactly one is installed, assume that is default, pipe the rest of the commands into the openclaw cli (temporary pod, or the on-host way we have now). It needs to be able to speak to the openclaw gateway. -It needs to be coming from an IP that openclaw will accept for security reasons. -[I guess this depends on what part of the code writes the skill files. If its the CLI, then these files would appear on the host, and we’d be back to packaging them like i would like to avoid.] -1. We could exec on the openclaw container itself and do everything local to the container runtime, that should sort auth and folder writing perms? -2. Plan b, we could on the host write to: $HOME/.config/obol/applications/obol/openclaw/playful-rabbit/.openclaw/skills/ and rely on openclaw’s hot reload behaviour -If more than one instance is installed, then we have to interpret the next word of command input as a petname, use it to decide the host path to write the skill to, or the correct gateway to kubectl exec on, before giving the remainder to a correctly configured openclaw cli (if needed) - -Pre-installed skills: -We need: -[Must have] Ethereum Network (erpc) -[Nice to have] Ethereum Wallet (web3signer) -[Next Release] Obol Doctor (debugs the obol stack) (also will be an obol doctor) -[A plan for a skill to sensibly deploy stuff into the stack] -Open question: -How does LiteLLM handle the idea of telling llms about tools they can opt to use? Is it just dealt with at the openclaw layer and not a concern of the llm? -Same problem as always, how do we get this folder structure on the disk for the openclaw instance. -We can start with: -Embedding the skills in obol-cli (golang) and copying them to the appropriate host directory upon creation/import of a new openclaw instance -Import should be cognizant of folder conficts in case this is an imported openclaw. -Will we have an obol/ namespace in ~/.openclaw/skills? is that standard? - -Nice to haves / hardening: -We should see if we can get the sandboxed skills feature to work -Let people turn off telemetry? (for now we want plenty though to show adoption). - - - -Telemetry (install counts) -When you run clawhub sync while logged in, the CLI sends a minimal snapshot to compute install counts. You can disable this entirely: -export CLAWHUB_DISABLE_TELEMETRY=1 - - - - -https://docs.openclaw.ai/tools/skills-config#skills-config - - -_______________ [Opus 4.6] _______ - -# Skills System Redesign: Remote Skills & Default Skill Packs - -## Problem Statement - -OpenClaw skills are currently filesystem-based directories with a `SKILL.md` file. The only way to get custom skills into a deployed OpenClaw instance is: - -1. Have them locally on disk -2. Run `obol openclaw skills sync --from ` to package them into a ConfigMap -3. Run `obol openclaw sync ` to apply - -There is no way to: -- Pull skills from public GitHub repositories -- Share skills across the community -- Ship default Obol-specific skills with the stack -- Manage skill dependencies or versions -- Add/remove individual skills without replacing the entire set - -## Goals - -1. **`npx`-like skill fetching** - `obol openclaw skills add github.com/org/repo` pulls skills from GitHub -2. **Default Obol skill pack** - ships useful Obol Stack skills out of the box (k8s ops, network management, monitoring) -3. **Skill registry resolution** - shorthand like `@obol/ethereum` resolves to a known GitHub location -4. **Declarative skill management** - skills can be listed in config for reproducible setups -5. **Backward compatible** - existing `skills sync --from ` still works - -## Current Architecture - -### How OpenClaw Loads Skills (npm package internals) - -``` -Load precedence (highest → lowest): -1. /skills/ — per-agent workspace skills -2. ~/.openclaw/skills/ — managed/local skills -3. Bundled skills (npm package) — 40+ built-in skills -4. skills.load.extraDirs — additional paths from openclaw.json -``` - -Each skill is a directory containing `SKILL.md` with YAML frontmatter: - -```markdown ---- -name: my-skill -description: What it does -metadata: - openclaw: - requires: - bins: ["kubectl"] - env: ["KUBECONFIG"] ---- - -# Agent instructions for using this skill... -``` - -### How Obol Stack Delivers Skills Today - -``` -obol openclaw skills sync --from - │ - ├─ tar -czf skills.tgz -C . - ├─ kubectl delete configmap openclaw--skills (if exists) - ├─ kubectl create configmap openclaw--skills --from-file=skills.tgz= - └─ prints "To apply, re-sync: obol openclaw sync " -``` - -The Helm chart (remote `obol/openclaw v0.1.3`) mounts this ConfigMap and extracts it into the pod's skills directory. - -### Overlay Values (current) - -```yaml -skills: - enabled: true - createDefault: true # chart creates empty ConfigMap placeholder -``` - -### Key Constraints - -- The Helm chart is **remote** (`obol/openclaw` from `obolnetwork.github.io/helm-charts/`), not in this repo ANSWER: You can update this chart, its adjacent to you in ../helm-charts. -- Skills ConfigMap has a **1MB limit** (etcd object size limit) — fine for text-based SKILL.md files but limits total skill count ANSWER: lets modify folders on localhost, which are mapped straight into the pods PVs, and openclaw runs a file watcher so it will just detect and reload -- The pod needs skills at filesystem paths — whatever we do must end up as files in the container -- OpenClaw's `skills.load.extraDirs` config and `skills.entries` per-skill config are available levers. ANSWER: and knowing the right host path to write to to end up at ~/.openclaw/skills - ---- - -## Proposed Design - -### Architecture Overview - -``` - ┌─────────────────────────────┐ - │ GitHub / Git Repositories │ - │ │ - │ github.com/ObolNetwork/ │ - │ openclaw-skills/ │ - │ github.com/user/ │ - │ my-custom-skill/ │ - └──────────┬──────────────────┘ - │ - ┌───────────────────┼───────────────────┐ - │ │ │ - ┌─────▼─────┐ ┌───────▼───────┐ ┌──────▼──────┐ - │ CLI Fetch │ │ Init Container│ │ Declarative│ - │ (dev UX) │ │ (GitOps) │ │ Config │ - └─────┬─────┘ └───────┬───────┘ └──────┬──────┘ - │ │ │ - ▼ ▼ ▼ - ┌──────────────────────────────────────────────────┐ - │ Local Skills Directory │ - │ $CONFIG_DIR/applications/openclaw//skills/ │ - │ │ - │ ├── @obol/ │ - │ │ ├── kubernetes/SKILL.md │ - │ │ ├── ethereum/SKILL.md │ - │ │ └── monitoring/SKILL.md │ - │ ├── @user/ │ - │ │ └── custom-skill/SKILL.md │ - │ └── skills.lock.json │ - └──────────────────┬───────────────────────────────┘ - │ - │ obol openclaw skills sync - │ (tar → ConfigMap → helmfile sync) - ▼ - ┌──────────────────┐ - │ OpenClaw Pod │ - │ /skills/ mount │ - └──────────────────┘ -``` - -### Component 1: Skill Source Resolution (`internal/openclaw/skills/`) - -A new `skills` subpackage that handles fetching skills from various sources. - -#### Source Types - -```go -// SkillSource represents a fetchable skill location -type SkillSource struct { - Type string // "github", "local", "builtin" - Owner string // GitHub org/user - Repo string // Repository name - Path string // Subdirectory within repo (optional) - Ref string // Git ref: tag, branch, commit (default: HEAD) - Alias string // Local name override -} -``` - -#### Resolution Rules - -| Input | Resolves To | -|-------|-------------| -| `@obol/kubernetes` | `github.com/ObolNetwork/openclaw-skills/skills/kubernetes@latest` | -| `@obol/ethereum` | `github.com/ObolNetwork/openclaw-skills/skills/ethereum@latest` | -| `github.com/user/repo` | Clone entire repo, find all `SKILL.md` files | -| `github.com/user/repo/path/to/skill` | Clone repo, use specific subdirectory | -| `github.com/user/repo@v1.2.0` | Clone at specific tag | -| `./local/path` | Copy from local filesystem (existing behavior) | - -#### Registry File - -A simple JSON registry embedded in the obol CLI binary that maps shorthand names to GitHub sources: - -```go -//go:embed skills-registry.json -var defaultRegistry []byte -``` - -```json -{ - "version": 1, - "prefix": "@obol", - "repository": "github.com/ObolNetwork/openclaw-skills", - "skills": { - "kubernetes": { - "path": "skills/kubernetes", - "description": "Kubernetes cluster operations via kubectl", - "requires": { "bins": ["kubectl"] } - }, - "ethereum": { - "path": "skills/ethereum", - "description": "Ethereum node management and monitoring", - "requires": { "bins": ["kubectl"] } - }, - "monitoring": { - "path": "skills/monitoring", - "description": "Prometheus/Grafana monitoring operations" - }, - "network-ops": { - "path": "skills/network-ops", - "description": "Obol network install/sync/delete operations" - }, - "tunnel": { - "path": "skills/tunnel", - "description": "Cloudflare tunnel management" - } - } -} -``` - -### Component 2: CLI Commands (`cmd/obol/openclaw.go`) - -Expand the `skills` subcommand group: - -``` -obol openclaw skills -├── add [--ref ] # Fetch skill(s) from GitHub or local path -├── remove # Remove an installed skill -├── list [--remote] # List installed skills (or available @obol skills) -├── sync # Push local skills dir → ConfigMap → pod -├── update [|--all] # Update skill(s) to latest version -└── init [--defaults] # Initialize skills dir with default Obol pack -``` - -#### `obol openclaw skills add` — the npx-like command - -```bash -# Add from the Obol registry (shorthand) -obol openclaw skills add @obol/kubernetes -obol openclaw skills add @obol/ethereum @obol/monitoring - -# Add from any public GitHub repo -obol openclaw skills add github.com/someuser/cool-skill -obol openclaw skills add github.com/someuser/skill-pack/skills/specific-one - -# Add from GitHub with version pinning -obol openclaw skills add github.com/someuser/cool-skill@v2.0.0 - -# Add from local directory (replaces old --from behavior) -obol openclaw skills add ./my-local-skills/custom-skill - -# Add all default Obol skills -obol openclaw skills add @obol/defaults -``` - -**Flow:** - -``` -obol openclaw skills add @obol/kubernetes - │ - ├─ Resolve "@obol/kubernetes" → github.com/ObolNetwork/openclaw-skills/skills/kubernetes - ├─ Sparse checkout (or GitHub API tarball) of just that path - ├─ Validate: SKILL.md exists with valid frontmatter - ├─ Copy to: $CONFIG_DIR/applications/openclaw//skills/@obol/kubernetes/ - ├─ Update skills.lock.json with source, ref, commit SHA - ├─ Print: "✓ Added @obol/kubernetes" - └─ Print: "Run 'obol openclaw skills sync ' to deploy" -``` - -#### `obol openclaw skills init` — bootstrap with defaults - -```bash -# Initialize with the default Obol skill pack -obol openclaw skills init default --defaults - -# This is equivalent to: -obol openclaw skills add @obol/defaults -obol openclaw skills sync default -``` - -#### `obol openclaw skills list` - -```bash -$ obol openclaw skills list default -Installed skills for openclaw/default: - - @obol/kubernetes Kubernetes cluster operations v1.0.0 (up to date) - @obol/ethereum Ethereum node management v1.0.0 (up to date) - @obol/monitoring Prometheus/Grafana operations v1.0.0 (update: v1.1.0) - custom-skill My custom skill from local local - -Total: 4 skill(s) - -$ obol openclaw skills list --remote -Available skills from @obol registry: - - @obol/kubernetes Kubernetes cluster operations via kubectl - @obol/ethereum Ethereum node management and monitoring - @obol/monitoring Prometheus/Grafana monitoring operations - @obol/network-ops Obol network install/sync/delete operations - @obol/tunnel Cloudflare tunnel management -``` - -### Component 3: Skills Lock File - -Track installed skills and their versions for reproducibility: - -```json -{ - "version": 1, - "skills": { - "@obol/kubernetes": { - "source": "github.com/ObolNetwork/openclaw-skills", - "path": "skills/kubernetes", - "ref": "v1.0.0", - "commit": "abc123def456", - "installed": "2026-02-18T12:00:00Z" - }, - "@obol/ethereum": { - "source": "github.com/ObolNetwork/openclaw-skills", - "path": "skills/ethereum", - "ref": "v1.0.0", - "commit": "abc123def456", - "installed": "2026-02-18T12:00:00Z" - }, - "custom-skill": { - "source": "local", - "path": "/Users/dev/my-skills/custom-skill", - "installed": "2026-02-18T14:00:00Z" - } - } -} -``` - -### Component 4: GitHub Fetching Strategy - -Two approaches, use **GitHub API tarball** as primary (no git dependency): - -```go -// Primary: GitHub API tarball download (no git required) -func fetchFromGitHub(owner, repo, path, ref string) (string, error) { - // GET https://api.github.com/repos/{owner}/{repo}/tarball/{ref} - // Extract only the files under {path}/ - // Return path to extracted directory -} - -// Fallback: git sparse-checkout (for private repos or rate limiting) -func fetchViaGit(repoURL, path, ref string) (string, error) { - // git clone --depth 1 --filter=blob:none --sparse - // git sparse-checkout set - // Return path to checked out directory -} -``` - -**Rate limiting**: GitHub API allows 60 requests/hour unauthenticated, 5000 with a token. For the `add` command this is fine (one request per skill add). Support `GITHUB_TOKEN` env var for authenticated requests. - -### Component 5: Default Skills in Onboard Flow - -Modify `Onboard()` to optionally install default skills: - -```go -// In Onboard(), after writing overlay and helmfile: -if opts.Sync { - // Install default Obol skills if skills dir is empty - skillsDir := filepath.Join(deploymentDir, "skills") - if _, err := os.Stat(skillsDir); os.IsNotExist(err) { - fmt.Println("Installing default Obol skills...") - if err := installDefaultSkills(skillsDir); err != nil { - fmt.Printf("Warning: could not install default skills: %v\n", err) - // Non-fatal — continue with deployment - } - } - // Skills sync happens as part of doSync -} -``` - -The default skills should be fetched from `@obol/defaults` (which maps to a curated list). If network is unavailable, fall back to a minimal embedded skill set. - -### Component 6: Embedded Fallback Skills - -For air-gapped or offline scenarios, embed a minimal set of skills directly in the CLI binary: - -```go -//go:embed skills/kubernetes/SKILL.md -//go:embed skills/network-ops/SKILL.md -var embeddedSkills embed.FS -``` - -These serve as a fallback when GitHub is unreachable during `skills init --defaults`. - -### Component 7: Overlay Values Enhancement - -Update `generateOverlayValues()` to support skill configuration in the Helm values: - -```yaml -skills: - enabled: true - createDefault: true - # NEW: Configure per-skill settings via overlay - entries: - kubernetes: - enabled: true - ethereum: - enabled: true - env: - ETHEREUM_NETWORK: "mainnet" - monitoring: - enabled: true -``` - -This maps to OpenClaw's `skills.entries` configuration, giving operators control over which skills are active and their per-skill environment. - -### Component 8: Automatic Skills Sync on Deploy - -Modify `doSync()` to automatically package and push skills if the local skills directory exists: - -```go -func doSync(cfg *config.Config, id string) error { - deploymentDir := deploymentPath(cfg, id) - - // Auto-sync skills if local skills directory exists - skillsDir := filepath.Join(deploymentDir, "skills") - if info, err := os.Stat(skillsDir); err == nil && info.IsDir() { - entries, _ := os.ReadDir(skillsDir) - // Only sync if there are actual skill directories (not just lock file) - hasSkills := false - for _, e := range entries { - if e.IsDir() { - hasSkills = true - break - } - } - if hasSkills { - fmt.Println("Syncing skills to cluster...") - if err := SkillsSync(cfg, id, skillsDir); err != nil { - fmt.Printf("Warning: skills sync failed: %v\n", err) - } - } - } - - // ... existing helmfile sync logic -} -``` - -This removes the two-step manual process. Adding a skill and syncing the deployment automatically picks it up. - ---- - -## Proposed Obol Default Skills - -These would live in `github.com/ObolNetwork/openclaw-skills`: - -### `@obol/kubernetes` - -```markdown ---- -name: kubernetes -description: Kubernetes cluster operations for the Obol Stack -metadata: - openclaw: - requires: - bins: ["kubectl"] - env: ["KUBECONFIG"] ---- - -# Kubernetes Operations - -You have access to kubectl configured for the Obol Stack k3d cluster. - -## Capabilities -- List, describe, and inspect pods, services, deployments across all namespaces -- View pod logs and events -- Check resource usage and node status -- Debug failing pods (describe, logs, events) - -## Conventions -- The stack uses k3d with namespaces per deployment -- Network deployments: `ethereum-`, `helios-`, `aztec-` -- Infrastructure: `traefik`, `erpc`, `monitoring`, `llm`, `obol-frontend` -- Use `kubectl get all -n ` for namespace overview -``` - -### `@obol/ethereum` - -```markdown ---- -name: ethereum -description: Ethereum node management and monitoring -metadata: - openclaw: - requires: - bins: ["kubectl"] ---- - -# Ethereum Node Management - -Manage Ethereum network deployments in the Obol Stack. - -## Capabilities -- Monitor execution and beacon client sync status -- Check peer counts and network connectivity -- View client logs for debugging -- Monitor disk usage and resource consumption -- Check chain head and sync progress - -## Common Operations -- Sync status: `kubectl -n ethereum- logs deploy/execution -f` -- Beacon status: `kubectl -n ethereum- logs deploy/beacon -f` -- Resource usage: `kubectl -n ethereum- top pods` -``` - -### `@obol/monitoring` - -```markdown ---- -name: monitoring -description: Prometheus and Grafana monitoring operations -metadata: - openclaw: - requires: - bins: ["kubectl"] ---- - -# Monitoring Operations - -Access Prometheus metrics and Grafana dashboards for the Obol Stack. - -## Capabilities -- Query Prometheus for metrics -- Check alerting rules and firing alerts -- Monitor resource usage trends -- Access Grafana dashboards -``` - -### `@obol/network-ops` - -```markdown ---- -name: network-ops -description: Obol network deployment lifecycle operations -metadata: - openclaw: - requires: - bins: ["kubectl"] ---- - -# Network Operations - -Manage the full lifecycle of blockchain network deployments. - -## Capabilities -- List installed network deployments -- Check deployment health and sync status -- Monitor resource consumption per deployment -- Assist with network configuration decisions -``` - -### `@obol/tunnel` - -```markdown ---- -name: tunnel -description: Cloudflare tunnel management for public access -metadata: - openclaw: - requires: - bins: ["kubectl"] ---- - -# Tunnel Management - -Manage Cloudflare tunnels for exposing Obol Stack services publicly. - -## Capabilities -- Check tunnel status and connectivity -- View tunnel logs for debugging -- Monitor tunnel routes and DNS configuration -``` - ---- - -## Implementation Phases - -### Phase 1: Core Skill Fetching (MVP) - -**Files to create/modify:** - -| File | Action | Description | -|------|--------|-------------| -| `internal/openclaw/skills/resolve.go` | Create | Source resolution (GitHub URL parsing, @obol shorthand) | -| `internal/openclaw/skills/fetch.go` | Create | GitHub tarball download + extraction | -| `internal/openclaw/skills/lock.go` | Create | Lock file read/write | -| `internal/openclaw/skills/registry.go` | Create | Embedded registry loading | -| `internal/openclaw/skills/skills-registry.json` | Create | Default @obol skill registry | -| `cmd/obol/openclaw.go` | Modify | Add `skills add`, `skills remove`, `skills list`, `skills update` subcommands | -| `internal/openclaw/openclaw.go` | Modify | Update `SkillsSync` to work with new skills dir layout | - -**Deliverables:** -- `obol openclaw skills add ` works with GitHub URLs and @obol shorthand -- `obol openclaw skills remove ` removes a skill -- `obol openclaw skills list` shows installed skills -- Lock file tracks installed skills -- Existing `skills sync --from` still works - -### Phase 2: Default Skills & Auto-Install - -**Files to create/modify:** - -| File | Action | Description | -|------|--------|-------------| -| `internal/openclaw/skills/defaults.go` | Create | Default skill installation logic | -| `internal/openclaw/skills/embedded/` | Create | Minimal embedded fallback skills | -| `internal/openclaw/openclaw.go` | Modify | Wire default skills into `Onboard()` flow | -| `internal/openclaw/openclaw.go` | Modify | Auto-sync skills in `doSync()` | - -**Deliverables:** -- `obol openclaw skills init --defaults` bootstraps default skills -- `Onboard()` installs defaults on first deploy (with network fallback to embedded) -- `doSync()` automatically packages skills if present -- No more two-step manual skills sync - -### Phase 3: Skill Pack Repository - -**External repository:** `github.com/ObolNetwork/openclaw-skills` - -| Path | Description | -|------|-------------| -| `skills/kubernetes/SKILL.md` | K8s cluster operations | -| `skills/ethereum/SKILL.md` | Ethereum node management | -| `skills/monitoring/SKILL.md` | Prometheus/Grafana ops | -| `skills/network-ops/SKILL.md` | Network lifecycle management | -| `skills/tunnel/SKILL.md` | Cloudflare tunnel management | -| `README.md` | Contributing guide for community skills | - -**Deliverables:** -- Public repo with curated Obol skills -- CI validation that all skills have valid SKILL.md frontmatter -- Tagged releases for version pinning - -### Phase 4: Helm Chart Integration (Upstream) - -Changes to the **remote** `obol/openclaw` Helm chart (separate repo): - -- Support `skills.sources` in values for declarative skill fetching via init container -- Init container that can `git clone` or download skills from configured sources -- This enables GitOps workflows where skills are declared in values, not manually pushed - -```yaml -# Future values-obol.yaml -skills: - enabled: true - sources: - - name: obol-defaults - repo: github.com/ObolNetwork/openclaw-skills - ref: v1.0.0 - path: skills/ - - name: custom - repo: github.com/myorg/my-skills - ref: main - entries: - kubernetes: - enabled: true - ethereum: - enabled: true -``` - -This phase requires coordination with the upstream openclaw Helm chart maintainers. - ---- - -## Directory Layout (Post-Implementation) - -``` -$CONFIG_DIR/applications/openclaw// -├── values-obol.yaml -├── helmfile.yaml -├── values-obol.secrets.json -└── skills/ # NEW: managed skills directory - ├── skills.lock.json # Tracks sources, versions, commits - ├── @obol/ # Namespaced by source - │ ├── kubernetes/ - │ │ └── SKILL.md - │ ├── ethereum/ - │ │ └── SKILL.md - │ ├── monitoring/ - │ │ └── SKILL.md - │ ├── network-ops/ - │ │ └── SKILL.md - │ └── tunnel/ - │ └── SKILL.md - └── @someuser/ # Community skills - └── custom-skill/ - └── SKILL.md -``` - ---- - -## CLI UX Examples - -### First-time setup with defaults - -```bash -$ obol agent init -Generated deployment ID: default - ✓ Ollama detected at http://localhost:11434 - -✓ OpenClaw instance configured! - Installing default Obol skills... - ✓ Added @obol/kubernetes - ✓ Added @obol/ethereum - ✓ Added @obol/monitoring - ✓ Added @obol/network-ops - ✓ Added @obol/tunnel - -Deploying to cluster... - Syncing skills to cluster... - ✓ Skills ConfigMap updated: openclaw-default-skills - Running helmfile sync... - -✓ OpenClaw installed with 5 default skills! -``` - -### Adding a community skill - -```bash -$ obol openclaw skills add github.com/ethbuilder/validator-skill -Fetching github.com/ethbuilder/validator-skill... - ✓ Found valid SKILL.md (name: validator-ops, description: Ethereum validator management) - ✓ Added to skills/ethbuilder/validator-ops/ - -Run 'obol openclaw skills sync default' to deploy - -$ obol openclaw skills sync default -Syncing skills to cluster... - ✓ Skills ConfigMap updated: openclaw-default-skills - Running helmfile sync... -✓ Skills deployed (6 skills) -``` - -### Updating skills - -```bash -$ obol openclaw skills update --all -Checking for updates... - @obol/kubernetes v1.0.0 → v1.1.0 (updated) - @obol/ethereum v1.0.0 (up to date) - @obol/monitoring v1.0.0 → v1.0.1 (updated) - @obol/network-ops v1.0.0 (up to date) - @obol/tunnel v1.0.0 (up to date) - -Updated 2 skill(s). Run 'obol openclaw skills sync default' to deploy. -``` - ---- - -## Open Questions - -1. **ConfigMap size limit**: With many skills, we may hit the 1MB etcd limit. Should we split into multiple ConfigMaps or use a PVC-based approach for large skill sets? - -2. **Skill dependencies**: Should skills be able to declare dependencies on other skills? (e.g., `@obol/ethereum` depends on `@obol/kubernetes`). Adds complexity but prevents broken skill chains. - -3. **Private repository support**: Should we support `GITHUB_TOKEN` for private repos from day one, or add it later? The fetch code should accept it but the UX can wait. - -4. **Helm chart init container (Phase 4)**: This requires upstream chart changes. Should we propose the chart changes early and develop in parallel, or wait until the CLI-side is proven? - -5. **Skill validation**: Should `skills add` validate that the skill's `requires.bins` are available in the target pod image, or just warn? Strict validation prevents broken skills but may be too rigid. - -6. **Community skill registry**: Beyond `@obol/` shorthand, should there be a discoverable registry (like npm) for community skills? Or is GitHub search + convention (`openclaw-skill-*` repos) sufficient? - ---- - -## Risk Assessment - -| Risk | Probability | Impact | Mitigation | -|------|-------------|--------|------------| -| GitHub API rate limiting | Medium | Low | Support `GITHUB_TOKEN`, cache downloads, embedded fallback | -| ConfigMap size limit | Low | Medium | Monitor archive sizes, split if needed, document limits | -| Upstream chart incompatibility | Low | High | Test against pinned chart version, coordinate with chart maintainers | -| Stale/broken community skills | Medium | Low | Validation on `skills add`, clear error messages, `skills check` command | -| Network unavailable during init | Medium | Medium | Embedded fallback skills, graceful degradation | - ---- - -## Success Criteria - -- [ ] `obol openclaw skills add @obol/kubernetes` fetches and installs the skill in <5 seconds -- [ ] `obol agent init` installs default skills automatically on first deploy -- [ ] `obol openclaw skills list` shows all installed skills with version info -- [ ] Community skills from arbitrary GitHub repos work without special configuration -- [ ] Existing `skills sync --from ` workflow continues to work unchanged -- [ ] Default Obol skills provide meaningful agent capabilities for stack operations -- [ ] Skills survive pod restarts (ConfigMap-backed persistence) -- [ ] Lock file enables reproducible skill sets across environments - - diff --git a/plans/terminal-ux-improvement.md b/plans/terminal-ux-improvement.md deleted file mode 100644 index a331e7af..00000000 --- a/plans/terminal-ux-improvement.md +++ /dev/null @@ -1,135 +0,0 @@ -# Plan: Obol Stack CLI Terminal UX Improvement - -## Context - -The obol CLI (`cmd/obol`) and the bootstrap installer (`obolup.sh`) had inconsistent terminal output styles. obolup.sh had a clean visual language (colored `==>`, `✓`, `!`, `✗` prefixes, suppressed subprocess output), while the Go CLI used raw `fmt.Println` with no colors, no spinners, and direct subprocess passthrough that flooded the terminal with helmfile/k3d/kubectl output. Invalid commands produced poor error messages with no suggestions. - -**Goal**: Unify the visual language across both tools, capture subprocess output behind spinners, and add `--verbose`/`--quiet` flags for different user needs. - -**Decision**: User chose "Capture + spinner" for subprocess handling and Charmbracelet lipgloss as the styling library. - -## What Was Built - -### New Package: `internal/ui/` (7 files) - -| File | Exports | Purpose | -|------|---------|---------| -| `ui.go` | `UI` struct, `New(verbose)`, `NewWithOptions(verbose, quiet)` | Core type with TTY detection, verbose/quiet flags | -| `output.go` | `Info`, `Success`, `Warn`, `Error`, `Print`, `Printf`, `Detail`, `Dim`, `Bold`, `Blank` | Colored message functions matching obolup.sh's `log_*` style. Quiet mode suppresses all except Error/Warn. | -| `exec.go` | `Exec(ExecConfig)`, `ExecOutput(ExecConfig)` | Subprocess capture: spinner by default, streams with `--verbose`, dumps captured output on error | -| `spinner.go` | `RunWithSpinner(msg, fn)` | Braille spinner (`⠋⠙⠹⠸⠼⠴⠦⠧⠇⠏`) — minimal goroutine impl, no bubbletea | -| `prompt.go` | `Confirm`, `Select`, `Input`, `SecretInput` | Thin wrappers around `bufio.Reader` with lipgloss formatting | -| `errors.go` | `FormatError`, `FormatActionableError` | Structured error display with hints and next-step commands | -| `suggest.go` | `SuggestCommand`, `findSimilarCommands` | Levenshtein distance for "did you mean?" on unknown commands | - -### Output Style (unified across both tools) - -``` -==> Starting cluster... (blue, top-level header — no indent) - ✓ Cluster created (green, subordinate result — 2-space indent) - ! DNS config skipped (yellow, warning — 2-space indent) -✗ Helmfile sync failed (red, error — no indent) -``` - -### Subprocess Capture Pattern - -- **Default** (TTY, not verbose): Spinner + buffer. Success → ` ✓ msg (Xs)`. Failure → `✗ msg` + dump captured output. -- **`--verbose`**: Stream subprocess output live, each line prefixed with dim ` │ `. -- **Non-TTY** (pipe/CI): Plain text, no spinner, live stream. -- **Exception**: Passthrough commands (`obol kubectl`, `obol helm`, `obol k9s`, `obol openclaw cli`) keep direct stdin/stdout piping. - -### Global Flags - -| Flag | Env Var | Effect | -|------|---------|--------| -| `--verbose` | `OBOL_VERBOSE=1` | Stream subprocess output live with `│` prefix | -| `--quiet` / `-q` | `OBOL_QUIET=1` | Suppress all output except errors and warnings | - -### CLI Improvements - -- **Colored errors**: `log.Fatal(err)` replaced with `✗ error message` (red) -- **"Did you mean?"**: Levenshtein-based command suggestions on typos (`obol netwerk` → "Did you mean? obol network") -- **Interactive prompts**: `obol model setup` uses styled select menu + hidden API key input via `ui.SecretInput` - -## Phased Rollout (as executed) - -### Phase 1: Foundation -Created `internal/ui/` package (7 files), added lipgloss dependency, wired `--verbose` flag, `Before` hook, `CommandNotFound` handler, replaced `log.Fatal` with colored error output. - -**Files created**: `internal/ui/*.go` -**Files modified**: `go.mod`, `cmd/obol/main.go` - -### Phase 2: Stack Lifecycle (highest impact) -Migrated `stack init/up/down/purge` — the noisiest commands. Added `*ui.UI` to `Backend` interface. Converted ~8 subprocess passthrough sites to `u.Exec()`. `waitForAPIServer` and polling loops wrapped in spinners. - -**Files modified**: `internal/stack/stack.go`, `internal/stack/backend.go`, `internal/stack/backend_k3d.go`, `internal/stack/backend_k3s.go`, `internal/stack/backend_test.go`, `internal/stack/stack_test.go`, `cmd/obol/bootstrap.go`, `cmd/obol/main.go` - -### Phase 3: Network + OpenClaw + App + Agent -Migrated network install/sync/delete, openclaw onboard/sync/setup/delete/skills, app install/sync/delete, and agent init. Cascaded `*ui.UI` through all call chains. Converted confirmation prompts to `u.Confirm()`. - -**Files modified**: `internal/network/network.go`, `internal/openclaw/openclaw.go`, `internal/openclaw/skills_injection_test.go`, `internal/app/app.go`, `internal/agent/agent.go`, `cmd/obol/network.go`, `cmd/obol/openclaw.go`, `cmd/obol/main.go` - -### Phase 4: Update, Tunnel, Model -Migrated remaining internal packages. `update.ApplyUpgrades` helmfile sync captured. All tunnel operations use `u.Exec()` (except interactive `cloudflared login` and `logs -f`). `model.ConfigureLLMSpy` status messages styled. - -**Files modified**: `internal/update/update.go`, `internal/tunnel/tunnel.go`, `internal/tunnel/login.go`, `internal/tunnel/provision.go`, `internal/model/model.go`, `cmd/obol/update.go`, `cmd/obol/model.go`, `cmd/obol/main.go` - -### Phase 5: Polish -Added `--quiet` / `-q` global flag with `OBOL_QUIET` env var. Quiet mode suppresses all output except errors/warnings. Migrated `obol model setup` interactive prompt to use `ui.Select()` + `ui.SecretInput()`. Fixed `cmd/obol/update.go` to use `getUI(c)` instead of `ui.New(false)`. - -**Files modified**: `internal/ui/ui.go`, `internal/ui/output.go`, `cmd/obol/main.go`, `cmd/obol/update.go`, `cmd/obol/model.go` - -### Phase 6: obolup.sh Alignment -Aligned the bash installer's output to match the Go CLI's visual hierarchy: -- `log_success`/`log_warn` gained 2-space indent (subordinate to `log_info`) -- Banner replaced from Unicode box (`╔═══╗`) to ASCII art logo (matches `obol --help`) -- Added `log_dim()` function and `DIM`/`BOLD` ANSI codes -- Instruction blocks indented consistently (2-space for text, 4-space for commands) - -**Files modified**: `obolup.sh` - -## Dependencies Added - -``` -github.com/charmbracelet/lipgloss — styles, colors, NO_COLOR support, TTY degradation -``` - -Transitive: `muesli/termenv`, `lucasb-eyer/go-colorful`, `mattn/go-runewidth`, `rivo/uniseg`, `xo/terminfo`. `mattn/go-isatty` was already an indirect dep. - -## Files Inventory - -**New files (7)**: -- `internal/ui/ui.go` -- `internal/ui/output.go` -- `internal/ui/exec.go` -- `internal/ui/spinner.go` -- `internal/ui/prompt.go` -- `internal/ui/errors.go` -- `internal/ui/suggest.go` - -**Modified Go files (~25)**: -- `go.mod`, `go.sum` -- `cmd/obol/main.go`, `cmd/obol/bootstrap.go`, `cmd/obol/network.go`, `cmd/obol/openclaw.go`, `cmd/obol/model.go`, `cmd/obol/update.go` -- `internal/stack/stack.go`, `internal/stack/backend.go`, `internal/stack/backend_k3d.go`, `internal/stack/backend_k3s.go` -- `internal/network/network.go` -- `internal/openclaw/openclaw.go` -- `internal/app/app.go` -- `internal/agent/agent.go` -- `internal/update/update.go` -- `internal/tunnel/tunnel.go`, `internal/tunnel/login.go`, `internal/tunnel/provision.go` -- `internal/model/model.go` -- `internal/stack/backend_test.go`, `internal/stack/stack_test.go`, `internal/openclaw/skills_injection_test.go` - -**Modified shell (1)**: -- `obolup.sh` - -## Verification - -1. `go build ./...` — compiles clean -2. `go vet ./...` — no issues -3. `go test ./...` — all 7 test packages pass -4. `bash -n obolup.sh` — syntax valid -5. `obol netwerk` — shows "Did you mean? obol network" -6. `obol --quiet network list` — suppresses output -7. `obol network list` — shows colored output with bold headers -8. `obol app install` — shows colored `✗` error with examples