feat: port `cli_dispatch.py` from `subprocess(claude -p)` to the Claude Agent SDK

[sriumcp]

TL;DR

Replace the current subprocess.Popen(["claude", "-p", …]) invocation with the official Claude Agent SDK. This is the foundational change — most of the other issues in this initiative become trivial once it lands.

Why this matters

Nous currently shells out to the Claude Code CLI in headless mode (orchestrator/cli_dispatch.py:219-222):

claude -p --model <model> --output-format json --dangerously-skip-permissions --max-turns <turns>

That works, but it's a thin shim with significant downsides:

• No native streaming. The orchestrator is blind for up to 7200 seconds. The 5/18 sessions show ~90-minute waits with zero progress signal — directly responsible for the "report progress" tic.
• No programmatic prompt caching. prompts/methodology/design.md (266 lines) and execute_analyze.md (199 lines) are static and re-sent uncached on every call.
• No native subagent spawning. Parallelizing arms (issue Framework buildout: 4-layer approach from methodology docs to autonomous orchestrator #3) requires custom worktree + subprocess fan-out; the SDK gives this for free.
• Retry semantics by subprocess restart. When a connection drops mid-turn, the message context is lost and the next attempt has to rebuild it. The SDK retries with full message history.
• Custom JSON parser to maintain. cli_dispatch.py has a hand-rolled parser for --output-format json plus error-classification heuristics.

What's already shipped this builds on

• feat: retry transient claude -p subprocess errors with exponential backoff #71 (retry on transient errors), feat: pre-flight check + retry everything with failure persistence #111 (pre-flight + persistence), fix: resume mid-flight campaign at correct iteration after timeout #91 (resume mid-flight). The retry primitives become simpler under the SDK because they operate on messages, not subprocesses.

Proposed approach

1. Add claude-agent-sdk (or whichever Anthropic SDK package is current at the time) to pyproject.toml.
2. Create orchestrator/sdk_dispatch.py mirroring the LLMDispatcher interface (same dispatch() signature, same artifact contract).
3. Use the SDK's session API to drive a single agentic loop per phase. Pass system blocks (cached in Agentic Systems Experimentation and Discovery (ASED) #2) and user messages (variable).
4. Migrate token/cost telemetry (feat: add token/cost tracking to both dispatchers #41) to read from SDK usage events directly; keep llm_metrics.jsonl schema for backward compatibility.
5. Add --agent sdk to the CLI alongside the existing api and inline options. Promote sdk to default after a release of real-world testing.
6. Delete CLI-specific code paths in a follow-up PR once SDK is the default.

Acceptance criteria

• nous run --agent sdk campaign.yaml runs a campaign end-to-end with parity on the existing test suite.
• Token telemetry shows native cache_read counters (set up in Agentic Systems Experimentation and Discovery (ASED) #2).
• cli_dispatch.py is unchanged but no longer the default path.
• Documentation in docs/architecture.md updated.

Out of scope

• Removing cli_dispatch.py entirely (do that in a later cleanup PR).
• Anything that depends on subagents, hooks, or caching — covered by Agentic Systems Experimentation and Discovery (ASED) #2, Framework buildout: 4-layer approach from methodology docs to autonomous orchestrator #3, Phase 2: Agent prompts + review protocol #8, Phase 4: Multi-iteration campaigns #9, Add docs/contributing/workflow.md: Claude-based PR creation workflow #12.

---

Part of #120.

[sriumcp]

Closed by #153 (squash-merge 3e69b85, merged into reflective 2026-05-24). The merged commit message contains an explicit Closes #121. GitHub auto-close didn't fire because the PR targets reflective rather than the repo's default branch.