Skip to content

Commit c8edf34

Browse files
committed
docs: refresh docs to match the now-shipped reality (real swarm, Gold Seal, honest pipeline, live model)
Audited the tracked product docs against what is actually on `main` and applied surgical, honest fixes. Already-accurate docs (korg-registry, korg-embeddings, korg-auth, korg-bridge, korg-seal) were left untouched. README.md: - Status section rewritten: honest shipped/planned split; 300+ tests (was a stale "175"); swarm listed as a real 5-persona swarm; surfaced the Gold Seal, honest pipeline, live-model and in-browser verifiers. - BERT/Candle made honest (optional `candle` feature; deterministic fallback). - Comparison gained Gold Seal + honest-attestation rows; added the live-model swarm example + in-browser verifier links; removed stale "ARCHITECTURE coming soon". ARCHITECTURE.md: 4-persona -> 5-persona (added the distinct `Persona::Evaluator`); new "Verifiable Flight-Recorder Capabilities" section (ledger@v1, Gold Seal, honest run-once, provider model) with a clearly-marked Planned list. ROADMAP.md: removed micro-healing from future work (it is already shipped). Crate/adapter READMEs + docs/: removed invented perf/size numbers (GIT_ISOLATION "<50ms", POSITIONING "200-line"); de-stale'd brittle exact test counts (korg-core, korg-setup) or corrected them to the real counts (recall-mcp); documented `response_format` + `DeterministicProvider` (korg-llm); clarified legacy flat-ledger vs the recommended verifiable per-session capture (claude-code); honest "no bundled SPA/WASM frontend" (korg-server); corrected stale placeholder/stub claims that no longer match the code (korg-tui). Each edit was audited against the source and re-checked by an adversarial honesty critic; spot-verified that all numbers are counted, not invented.
1 parent b85820b commit c8edf34

13 files changed

Lines changed: 119 additions & 65 deletions

File tree

ARCHITECTURE.md

Lines changed: 19 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ The entire Korg backend is implemented in Rust, which provides compile-time guar
1212

1313
- **Ownership and Borrowing**: Rust's ownership model ensures that there is always one and only one "owner" of any given piece of memory. Data can be immutably borrowed by multiple parties or mutably borrowed by exactly one party. This statically prevents data races at compile time. In Korg, shared state like the `Blackboard` is wrapped in concurrency primitives like `Arc<Mutex<T>>`, which enforce these borrowing rules at runtime, ensuring that even in a highly concurrent multi-agent system, there are no data races.
1414
- **Absence of Null and Undefined Behavior**: Rust's type system eliminates null pointer dereferences through the `Option<T>` and `Result<T, E>` enums. This forces developers to handle the absence of a value explicitly, preventing entire classes of bugs and security vulnerabilities common in other systems languages.
15-
- **Fearless Concurrency**: The combination of ownership rules and type-system constraints allows Korg to manage the 4-Persona Swarm with "fearless concurrency." The compiler guarantees that any data shared between threads (i.e., between the Leader orchestrator and its various tasks) is done so safely.
15+
- **Fearless Concurrency**: The combination of ownership rules and type-system constraints allows Korg to manage the 5-Persona Swarm with "fearless concurrency." The compiler guarantees that any data shared between threads (i.e., between the Leader orchestrator and its various tasks) is done so safely.
1616

1717
### 1.2. Isolated Execution via Process Sandboxing
1818

@@ -34,17 +34,17 @@ At the heart of Korg's concurrency model is the **Blackboard**, a central, obser
3434
- **CRDT-like Merging**: The `perform_semantic_merge` function in `src/leader.rs` exemplifies this principle. It takes the outputs from all competing personas and uses a "Synthesizer" persona (Lucas) to reconcile them into a single, cohesive set of mutations. This is analogous to a CRDT merge function that resolves concurrent updates into a deterministic, final state.
3535
- **Observability**: The Blackboard is the "Gravitational Well" around which the swarm orbits. It is the single source of truth for system telemetry. Personas emit `SwarmTelemetryPulse` messages, which the Leader ingests into the Blackboard's `trace_buffer`. The `Evaluator` then reads this buffer to assess the swarm's health, creating a tight, observable feedback loop. The state is exposed via the `/api/state` endpoint for the web cockpit, providing real-time observability to the human operator.
3636

37-
## 3. 4-Persona Adversarial Swarm Topology
37+
## 3. 5-Persona Adversarial Swarm Topology
3838

39-
Korg employs a fixed 4-persona swarm, where each agent has a specialized, adversarial role. This division of labor creates a system of checks and balances that promotes robust, high-quality output.
39+
Korg employs a fixed 5-persona swarm, where each agent has a specialized, adversarial role. This division of labor creates a system of checks and balances that promotes robust, high-quality output.
4040

4141
1. **Orchestrator Captain (The Planner)**:
4242
- **Role**: High-level strategist and planner.
4343
- **Function**: Receives the initial root task from the operator. Decomposes the task into smaller, actionable work packages (`decompose_into_persona_packages` in `src/leader.rs`). Initiates the `ContractNegotiation` phase, proposing acceptance criteria for the task. The Captain sets the direction for the entire swarm.
4444

4545
2. **Auditor Harper (The Researcher/Critic)**:
4646
- **Role**: Context-gatherer, researcher, and adversarial critic.
47-
- **Function**: During `ContractNegotiation`, Harper (acting as part of the `Evaluator`) critiques the Captain's proposed criteria for ambiguity and relevance, forcing a more robust contract. During execution, Harper is assigned "Research" tasks, gathering information from the codebase or external sources to inform the Builder. Harper's primary function is to prevent the swarm from "hallucinating" or working with incorrect assumptions.
47+
- **Function**: During `ContractNegotiation`, Harper critiques the Captain's proposed criteria for ambiguity and relevance, forcing a more robust contract. During execution, Harper is assigned "Research" tasks, gathering information from the codebase or external sources to inform the Builder. Harper's primary function is to prevent the swarm from "hallucinating" or working with incorrect assumptions.
4848

4949
3. **Builder Benjamin (The Coder)**:
5050
- **Role**: Primary code generator and implementer.
@@ -54,6 +54,10 @@ Korg employs a fixed 4-persona swarm, where each agent has a specialized, advers
5454
- **Role**: Merger and finalizer.
5555
- **Function**: After all personas have submitted their work, the `LeaderOrchestrator` invokes Lucas. As shown in `perform_semantic_merge`, Lucas's job is to take the winning proposal from the **Arena** and intelligently merge it with complementary ideas from the other personas. This produces a final, cohesive patch that is more robust than any single persona's output.
5656

57+
5. **Evaluator (The Harsh Adversarial Critic)**:
58+
- **Role**: Independent scoring authority and Generator/Evaluator loop.
59+
- **Function**: Spawned separately by the Leader (`Spawning Evaluator to score persona ...`), the Evaluator receives a generator persona's output and scores it against the negotiated contract rubrics in the **Arena**, following the Anthropic-style Generator/Evaluator loop. It is a distinct persona (`Persona::Evaluator`), not a role played by any of the four workers above, and it is what turns the swarm's output into a measured, adversarially-checked verdict.
60+
5761
This adversarial collaboration ensures that a plan is critiqued before it's executed, code is generated based on a solid plan, and multiple proposed solutions are synthesized into a superior final product.
5862

5963
## 4. State Transition Sequence
@@ -143,6 +147,17 @@ Korg maintains a tamper-proof audit trail of every significant action using a cr
143147
- Simultaneously, it uses the `state_merkle_root` to find the corresponding state blob and rehydrate the logical `Blackboard`.
144148
- This powerful feature allows an operator to rewind the AI's "thought process" and codebase to any valid, signed point in history and "fork" its execution in a new direction, all with cryptographic guarantees of integrity.
145149

150+
### 5.1. Verifiable Flight-Recorder Capabilities
151+
152+
Beyond the in-process provenance chain above, Korg ships a set of independently-verifiable, "flight-recorder" capabilities that let third parties check an agent's work without trusting Korg itself:
153+
154+
- **`korg-ledger@v1`**: A frozen, hash-chained, HLC-ordered, tamper-evident event ledger with cross-language conformance (Rust/Python/JS). Each event carries an Ed25519 per-event signature, and the ledger supports git-tip structural anchoring for offline structural verification.
155+
- **Gold Seal (`goldseal@v1`)**: A public, independently-verifiable certificate of agent work, produced by the `korg-seal` CLI (mint/anchor/resolve/verify) and checkable by three conformant verifiers (`korg-verify` in Rust, plus Python and JS) as well as zero-install in-browser verifiers hosted on GitHub Pages.
156+
- **Honest `korg run-once` pipeline**: Runs a real patch through a real `cargo check` and attests a mutation count that *equals* the real `git diff` (file count + changed paths). When it cannot produce a real result, it reports an honest null rather than fabricating one.
157+
- **Provider model**: The default is a hermetic `DeterministicProvider` (fixture-only); `--provider ollama` runs a live local model on arbitrary tasks.
158+
159+
**Planned (not yet shipped):** live *network* time-witness anchoring (the anchor structure and offline verification are shipped; only the trusted-time witness fetch remains an honest limit), `crates.io`/npm publishing of the CLIs and verifiers, and a `korg fork` / execution-checkpoint-restore CLI (the primitives exist; the shipped reversibility today is `korg rewind`).
160+
146161
## 6. OCR Pixel Redaction & Visual Firewall (`src/vision_policy.rs`)
147162

148163
Since Korg personas can interact with GUIs and web browsers, a critical security vector is the accidental exposure of sensitive data in screenshots. The Visual Policy Engine in `src/vision_policy.rs` acts as a fail-secure firewall for all visual data.

README.md

Lines changed: 37 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -178,6 +178,12 @@ korg campaign --web --prompt "Optimize the database connection pool"
178178
# Pure autonomous goal mode (--goal is a top-level flag)
179179
korg --goal "Write and validate a full test suite for src/parser.rs"
180180

181+
# Run the full multi-persona swarm on a REAL local model — every persona
182+
# (Captain, Harper, Benjamin, Lucas, Evaluator) runs as a real worker
183+
# subprocess doing real, measured, attested work. Defaults to a hermetic
184+
# deterministic provider; `--provider ollama` makes it live.
185+
korg --goal "Fix the failing test in src/lib.rs" --provider ollama --model qwen2.5:7b
186+
181187
# Preview without committing (dry-run; --preview is a top-level flag)
182188
korg --preview "Refactor the main event loop"
183189
```
@@ -216,6 +222,14 @@ korg-verify <path-to-ledger.jsonl>
216222
> pipeline cannot attest a number the worktree does not actually show — that is
217223
> the guarantee, independent of model quality.
218224
225+
> **Verify it in your browser — sends nothing.** Zero-install, client-side
226+
> verifiers (Web Crypto) for any `korg-ledger@v1` journal or Gold Seal:
227+
> [verify a session](https://new1direction.github.io/korg/web/index.html) ·
228+
> [verify a Gold Seal](https://new1direction.github.io/korg/web/seal.html) ·
229+
> [time-travel explorer](https://new1direction.github.io/korg/web/explore.html).
230+
> They hash-chain, check the causal DAG, validate Ed25519 signatures, and
231+
> re-derive the human summary from the events — all locally.
232+
219233
> Speculative branch/fork and named checkpoints (`korg fork`, `korg checkpoints
220234
> list|restore`) are planned, not yet shipped. The reversibility surface today is
221235
> `korg rewind`.
@@ -267,6 +281,8 @@ Korg treats AI cognition the same way a hypervisor treats compute and Git treats
267281
| Speculative branches | 🚧 planned ||||
268282
| Execution checkpoints | 🚧 planned ||||
269283
| Cryptographic audit trail |||||
284+
| Independently-verifiable Gold Seal |||||
285+
| Honest attestation (real diff, never fabricated) |||||
270286
| Micro-healing |||||
271287
| Model-agnostic |||||
272288

@@ -282,7 +298,7 @@ Korg treats AI cognition the same way a hypervisor treats compute and Git treats
282298
| Ledger ordering | Hybrid Logical Clocks (HLC) |
283299
| Workspace snapshots | Git Merkle tree (O(1) restore via `write-tree` / `read-tree`) |
284300
| Cryptographic attestation | Ed25519 (ed25519-dalek) |
285-
| Semantic governance | BERT cosine similarity (Candle / Hugging Face) |
301+
| Semantic governance | BERT cosine similarity via the optional `candle` feature (Hugging Face); a deterministic embedding fallback runs when `candle` is not built |
286302
| TUI dashboard | Ratatui + Crossterm |
287303
| Web cockpit | Axum + SSE |
288304
| Syntax highlighting | Syntect + tree-sitter |
@@ -291,7 +307,7 @@ Korg treats AI cognition the same way a hypervisor treats compute and Git treats
291307

292308
## Architecture Deep Dive
293309

294-
**[Read the full technical write-up](https://github.com/New1Direction/korg/blob/main/ARCHITECTURE.md)** *(coming soon)*
310+
**[Read the full technical write-up](https://github.com/New1Direction/korg/blob/main/ARCHITECTURE.md)**
295311

296312
### Real-World Audit Ledger Example
297313
You can inspect a real-world cognitive audit ledger produced by Korg. This NDJSON file records a live session where Claude Code was prompted to call Korg's MCP tools to refactor a function and rename all call sites, capturing the full HLC causal graph and `actor_id` recorder metadata:
@@ -309,23 +325,28 @@ The short version:
309325

310326
## Status
311327

312-
Korg is in active development. Current test coverage: **175 tests, 0 failures** (162 cargo across 8 crates + 13 pytest for the PyO3 bridge).
328+
Korg is in active development, built on a **frozen `korg-ledger@v1` spec with cross-language conformance** (Rust + Python + JS). Test footprint: **300+ Rust tests across the workspace plus Python/JS conformance suites**, CI-gated (build · tests · cross-language oracle · differential fuzz) and green on `main`.
313329

314-
- [x] Append-only cognitive ledger with HLC ordering
330+
**Shipped:**
331+
- [x] Append-only, hash-chained cognitive ledger with HLC ordering
315332
- [x] Deterministic replay and projection rebuilds
333+
- [x] Reversible execution — rewind the ledger to any prior sequence point (tamper-evident `LedgerRewind`)
334+
- [x] Per-event Ed25519 signatures + structural anchoring (`korg-ledger@v1` §8)
335+
- [x] **Gold Seal (`goldseal@v1`)** — a public, independently-verifiable certificate of agent work, with zero-install in-browser verifiers
336+
- [x] **Honest pipeline** (`korg run-once`) — real patch → real `cargo check` → an attested mutation count that equals the real `git diff`; never fabricates (reports an honest null instead)
337+
- [x] **Live local model** (`--provider ollama`) — real per-persona work on arbitrary tasks
338+
- [x] **Multi-agent swarm** (Captain, Harper, Benjamin, Lucas, Evaluator) — genuine worker subprocesses doing real, measured, attested work with DAG data-flow between personas
339+
- [x] Zero-config Claude Code capture (PostToolUse/Stop hooks → verifiable per-session ledgers)
340+
- [x] Micro-healing effect layer · TUI dashboard + Web cockpit
341+
- [x] Cryptographic provenance attestation · single-authority CognitionMode governance
316342
- [x] Preview / dry-run mode (`--preview`)
317-
- [ ] Speculative warm-boot execution (in progress)
318-
- [ ] Execution checkpoints / restore CLI (primitive exists; CLI planned)
319-
- [x] Micro-healing effect layer
320-
- [x] Multi-agent swarm orchestration (Captain, Harper, Benjamin, Lucas)
321-
- [x] TUI dashboard + Web cockpit
322-
- [x] Cryptographic provenance attestation
323-
- [x] Single-authority CognitionMode governance
324-
- [ ] `cargo install korg` on crates.io
325-
- [ ] Remote swarm workers
326-
- [ ] WASM backends
327-
- [ ] IDE language server integration
328-
- [ ] Distributed checkpoint synchronization
343+
344+
**Planned / not yet shipped:**
345+
- [ ] Speculative branches / fork + execution-checkpoint restore CLI (primitives exist; CLI planned)
346+
- [ ] `cargo install korg` on crates.io · npm-published verifier
347+
- [ ] Live network anchoring resolver (trusted-time witness — the remaining honest limit)
348+
- [ ] Remote swarm workers · WASM backends · IDE language-server integration · distributed checkpoint sync
349+
- [ ] Fully passive capture without agent cooperation
329350

330351
---
331352

ROADMAP.md

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,6 @@ without human intervention.
1515
| Feature | Description |
1616
|---|---|
1717
| `korg autoheal` | Autonomous recovery loop: detect failure → identify root cause → rewind to last clean checkpoint → retry with corrected strategy |
18-
| Micro-healing at the effect layer | Per-tool failure recovery without full session rewind |
1918
| Incremental checkpoints | Background checkpoint compression — only delta since last checkpoint is stored |
2019
| Recovery confidence scoring | Each autoheal attempt is scored; low-confidence recoveries escalate to human approval |
2120
| Doom-loop circuit breaker | Hard limit on recursive recovery attempts with structured escalation |

adapters/claude-code/README.md

Lines changed: 24 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -55,17 +55,24 @@ graph identical in shape to what korgex's own agent loop produces.
5555

5656
## Two ways to use it
5757

58-
### 1. Live tail mode (recommended)
58+
> **Recommended live path: `korg-setup` + `korg-hook`.** The verifiable,
59+
> zero-config capture model (PostToolUse/Stop hook → per-session verifiable
60+
> ledger at `~/.korg/sessions/<id>.jsonl`) is described in
61+
> [What it produces](#what-it-produces). The `korg-ingest-claude --tail/--once`
62+
> CLI below still works, but it emits the **legacy un-chained flat ledger**;
63+
> run `korg-backfill --migrate-flat <path>` to upgrade an existing flat ledger
64+
> to the per-session format.
5965
60-
`korg follows you in real time` — the adapter watches
61-
`~/.claude/projects/**/*.jsonl` and streams new events into a korg
62-
ledger as Claude Code writes them. Install, run once, get a continuously-
63-
growing ledger forever:
66+
### Legacy flat-file tail (`korg-ingest-claude`)
67+
68+
The adapter watches `~/.claude/projects/**/*.jsonl` and streams new events into
69+
a **legacy flat (un-chained)** korg ledger as Claude Code writes them. Install,
70+
run once, get a continuously-growing flat ledger:
6471

6572
```bash
6673
pip install -e ./adapters/claude-code
6774

68-
# Watch ~/.claude/projects, append events to ~/.korg/claude-events.jsonl
75+
# Watch ~/.claude/projects, append events to ~/.korg/claude-events.jsonl (legacy flat ledger)
6976
korg-ingest-claude --tail
7077

7178
# Just print events to stdout (no file write):
@@ -83,18 +90,23 @@ The byte-offset state is persistent — kill the process and restart any
8390
time; it picks up exactly where it left off. The state file at
8491
`~/.korg/claude-tail-state.json` survives reboots.
8592

86-
### 2. One-shot backfill
93+
### One-shot backfill of history
94+
95+
For the existing pile of session files on your disk right now, use
96+
`korg-backfill` — it re-derives the **verifiable per-session ledgers** under
97+
`~/.korg/sessions/` for every historical session.
8798

88-
For the existing pile of session files on your disk right now:
99+
The legacy `korg-ingest-claude --once` still exists, but it writes the **legacy
100+
un-chained flat ledger** instead:
89101

90102
```bash
91-
korg-ingest-claude --once --out ~/.korg/claude-events.jsonl
92-
# [korg-ingest-claude] one-shot pass complete · files_active=42 events=128304 ...
103+
korg-ingest-claude --once --out ~/.korg/claude-events.jsonl # legacy flat ledger
104+
# [korg-ingest-claude] one-shot pass complete · files_active=N events=M ...
93105
```
94106

95107
Runs once, ingests everything you haven't ingested yet, exits.
96108

97-
### 3. Library usage
109+
### Library usage
98110

99111
For embedding into your own ledger plumbing:
100112

@@ -109,7 +121,7 @@ def emit(body: dict) -> int | None:
109121

110122

111123
# Single-session replay:
112-
adapter = ClaudeCodeAdapter(emit, source_agent="agent:claude-code@2.1.150")
124+
adapter = ClaudeCodeAdapter(emit, source_agent="agent:claude-code@2.1.0")
113125
with Path("~/.claude/projects/-Users-you-Documents-foo/abc.jsonl").expanduser().open() as f:
114126
stats = adapter.ingest(f)
115127

adapters/korg-setup/README.md

Lines changed: 3 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -109,16 +109,9 @@ you can preview every action before committing.
109109

110110
## Tests
111111

112-
45 tests cover:
113-
114-
- 14 for the `~/.claude.json` editor (idempotent register/remove,
115-
atomic backups, preservation of unrelated keys, env vars).
116-
- 13 for the launchd integration (plist shape, write idempotency,
117-
install/uninstall, platform gating, `is_loaded` parser).
118-
- 11 for the setup orchestrator (binary detection, dry-run, idempotency,
119-
ledger-dir creation, launchctl-failure reporting, no-daemon path).
120-
- 7 for the status reporter (empty install, partial install, loaded
121-
launchd detection, format rendering).
112+
A test suite covers the `~/.claude.json` editor, launchd integration,
113+
setup orchestrator, status reporter, discovery, bridge registration, and
114+
Claude settings.
122115

123116
Run them with the Korg workspace venv:
124117

adapters/recall-mcp/README.md

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -181,17 +181,19 @@ Result text format (one line per match, designed for direct LLM consumption):
181181

182182
## Tests
183183

184-
49 cover everything end-to-end:
184+
Tests cover everything end-to-end:
185185

186186
- 9 tests for the text flattener (per-event-type extraction, trimming, fallbacks).
187-
- 9 tests for the index (incremental load, malformed-line tolerance,
187+
- 11 tests for the index (incremental load, malformed-line tolerance,
188188
partial-line hold-back, multi-file, new-file pickup, triggered_by preservation).
189-
- 17 tests for the recall engine (substring + semantic, top_n, min_score,
189+
- 16 tests for the recall engine (substring + semantic, top_n, min_score,
190190
tool_filter, automatic refresh, auto-mode fallback, explicit-semantic
191191
raising without fastembed).
192-
- 14 tests for the MCP server (full initialize/list/call roundtrip,
192+
- 16 tests for the MCP server (full initialize/list/call roundtrip,
193193
unknown-method errors, notification handling, ping, tool_filter through
194194
the JSON-RPC surface, malformed-line tolerance, serve_stdio loop).
195+
- 17 tests for the introspect document (korg:introspect@v1 schema,
196+
callables, exit_codes, command_id stability).
195197

196198
Run them with the Korg workspace venv (which has fastembed):
197199

0 commit comments

Comments
 (0)