docs(plans): multi-node roadmap for the volume-permission direction

Records the options ladder for 'nodes joining my cluster' (far-future, not
scheduled) so near-term permission decisions don't foreclose it: the
host-FS-canonical coupling and its four conditions, the agent-homes-only
scope, what breaks on day one of a join (WaitForFirstConsumer + no
nodeSelector in any render), options 0-4 (home-node pattern -> NFS ->
distributed storage -> API-mediated access -> hybrid by data class), join
mechanics (native k3s server + remote agents; macOS stays k3d), and the
decisions that bind today: group-1000 sharing over render-time UID
matching, home-node pinning as a join prerequisite, inputs to API objects
per Remaining Debt.

Author: bussyjd

plans/volume-permission-hardening.md

@@ -96,3 +96,107 @@ to Kubernetes-native inputs without making editable skills/workspaces immutable:
 Until that migration is done, do not reintroduce global provisioner chown,
 root init containers, or UID 1000 alignment as the primary fix for local-path
 PVC permissions.
+
+## Roadmap: multi-node ("nodes joining my cluster")
+
+Not scheduled — far-future. Recorded here so near-term permission decisions
+don't foreclose it.
+
+### The coupling we accepted
+
+"Host FS is canonical" is an identity: one directory is simultaneously a
+plain host path the CLI writes (`SeedHostFiles`, `syncRuntimeFiles`, wallet
+staging, flow-16 host asserts) AND the backing store of a PVC a pod mounts.
+The identity holds only while ALL of these are true:
+
+1. The PV is path-addressable on a node (local-path/hostPath, not block or
+   network storage).
+2. The node's FS is the host's FS (k3d bind-mount of `OBOL_DATA_DIR`, or
+   native k3s where host == node).
+3. There is effectively one node, so "the node that has the directory" and
+   "the node the pod runs on" cannot diverge.
+4. Host and container share UID/GID semantics (same kernel; on macOS faked
+   by Docker Desktop file sharing).
+
+A second node breaks (2) and (3) independently. Every permission mechanism
+in this plan (group sharing, fsGroup walks, the removed chowns) operates on
+(4) and silently assumes (1)–(3).
+
+### Scope of the coupling — agent homes only
+
+| Data | Host access needed? |
+|---|---|
+| hermes-data / agent homes (config, SOUL.md, skills) | Yes — the product promise |
+| OpenClaw skills | Yes |
+| x402-buyer-state (consumed.json) | No — pod-private |
+| Chain data (reth, aztec) | No — pod-private |
+| Wallet keystores | Only at creation; Remaining Debt moves them to Secrets |
+
+Scaling out means shrinking the host-canonical surface to the first two
+rows, not replicating it onto every node.
+
+### What breaks on day one of a join (as of this writing)
+
+- `local-path` is `volumeBindingMode: WaitForFirstConsumer` and NO render
+  (hermes.go, agent_render.go, llm.yaml) sets a nodeSelector/affinity: a new
+  agent's pod can schedule on node B, the PV is provisioned on node B's
+  disk, while `SeedHostFiles` writes to `$DATA_DIR` on the home host. The
+  pod boots against an empty home with no error anywhere.
+- `ensureVolumeWritable` is a `docker exec` into the k3d node container —
+  no transport to a remote node (already early-returns on the k3s backend).
+- Existing PVs pin pods to their node forever via nodeAffinity, so wrong
+  first placements are sticky.
+
+### Options ladder (increasing decoupling)
+
+0. **Home-node pattern (recommended v1, prerequisite for any join path).**
+   Label the home node (`obol.org/home=true`), render nodeSelector into
+   every host-canonical workload (hermes master, CRD agents, litellm/buyer).
+   Joined nodes take stateless or pod-private work only: vLLM/Ollama
+   upstreams, network nodes (the biggest storage consumers, zero host
+   visibility needed), demo servers. Agents cannot migrate; home node is
+   the SPOF; cheap and non-breaking.
+1. **Host exports `$DATA_DIR` over NFS** (csi-driver-nfs), agent-home PVCs
+   become RWX network mounts; files still physically live on the host so
+   direct editing keeps working. `all_squash,anonuid=1000` solves ownership
+   flapping at the protocol level. Hard caveat: Hermes `state.db` is SQLite
+   — SQLite over NFS is a corruption hazard. Workable shape is inputs over
+   NFS + state.db on a node-local PV, which is already half of option 3.
+2. **Distributed storage (Longhorn/Rook).** Solves migration, destroys host
+   access entirely, far too heavy for local-first. Ruled out except as the
+   storage class users bring on managed k8s.
+3. **API-mediated host access (the #610 direction, revisited).** Inputs
+   (config, SOUL.md, skills, markers) as ConfigMaps/Secrets/OCI artifacts —
+   delivered to any node, checksum-rolled; state as pod-private PVs on any
+   provisioner; host access becomes a verb (`obol agent fs ls|cat|edit|cp`
+   over kubectl exec/cp or a sidecar) instead of a shared mount. Survives
+   arbitrary topology and PSS-restricted namespaces. Cost: live-editing a
+   skill needs a sync round-trip instead of `:w` — UX problem, solvable
+   with a `--watch` loop; this is why #610 was reverted and why it comes
+   back when (3) in the coupling list stops being true.
+4. **Hybrid by data class (target end-state).** Operator-authored inputs →
+   API objects; machine state → pod-private PVs; human-inspectable outputs
+   → `obol agent fs` or a write-once RWX exports share. Single-node k3d
+   keeps the local-path fast path as an optimization, not as the contract.
+
+### Join mechanics
+
+k3d `node create` only adds agent containers on the same host. The real
+multi-node story is the native k3s backend: Linux home runs `k3s server`
+(host FS == node FS, home-node pattern costs nothing), remote boxes join
+with `k3s agent --server ... --token ...`. macOS stays single-node k3d;
+remote GPU capacity is better reached as an external endpoint
+(`obol model setup custom --endpoint http://gpu-box:8000/v1`), which the
+stack already supports and which sidesteps this entire section.
+
+### Decisions binding today
+
+- Prefer pure group-1000 sharing (setgid dirs, g+rw, nobody chowns owners)
+  over render-time `os.Getuid()` UID matching: group sharing is
+  topology-neutral; UID matching bakes one machine's identity into
+  manifests and deepens the coupling.
+- Before any join path ships: home-node nodeSelector rendering + a
+  join-time preflight that names the host-canonical volumes pinned to the
+  home node.
+- Input migration to API objects proceeds on the Remaining Debt schedule
+  above; it is also the multi-node prerequisite.