Skip to content

Commit 1e63b85

Browse files
ualtinokalfonso-magic-context
ualtinok
and
alfonso-magic-context
committed
docs(spec): CK#1 request-level precision + MC SUBC module internals (spec #5)
CK Message (#1): fold SUBC's two non-blocking precision notes before the canonical blessing - (1) state the keystone invariant at REQUEST granularity (render operates on a request, not a lone message; property test checks at request level); (2) add section 2.1 placing request-level fields (tool declarations + model = render-config epoch HARD-bust markers pinned per session, NOT CK Message content; CK Message is the messages array only; response metadata out of scope). MC SUBC module internals (#5, NEW): the transform brain - the per-pass transform() contract #4 drives; always-full baseline (Edge A) with optional plugin-delta; consuming the cache-policy core as an in-process observer adapter (anchor-check-first, signal mapping, no own SOFT/HARD ladder); the frozen-set render/store layer (byte-complete units Leak-A, atomic version-stamped CAS Leak-B/D, render-on-bust-only); durable store.db = cache-DECISION state only + byte-identical restart recovery via (full array + durable frozen-set) anchor-checked replay (Edge B); per-session actor (Edge F); the harness-model codec boundary (MessageV2/AgentMessage <-> CK, owned here, binding to CK#1 5.9); historian/dreamer relocate to run as llm-runner sessions (orchestrated not hosted, writes ride the next bust); the survives/collapses/relocates split; daemon-down clean-error-no-half-write (Edge D); conformance checklist. Co-authored-by: Alfonso [Magic Context] <288211368+alfonso-magic-context@users.noreply.github.com>
1 parent 1240e7d commit 1e63b85

2 files changed

Lines changed: 319 additions & 6 deletions

File tree

‎docs/specs/ck-message.md‎

Lines changed: 27 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -46,18 +46,39 @@ Two decoder families exist (codec spec #2 owns the detail):
4646
projection**. It drops all harness-only flags (bucket `flag`).
4747
- **`render(canonical)`** yields provider wire bytes.
4848

49-
**Keystone conformance invariant (NORMATIVE, checkable):**
49+
**Keystone conformance invariant (NORMATIVE, checkable).** `render()` operates on
50+
a REQUEST (a `Vec<CkMessage>` plus the render-config inputs, §2.1), so the
51+
invariant is stated at request granularity (the per-message form is the special
52+
case):
5053

5154
```
52-
for every CK message ck and every mutation m over harness-only flags:
53-
render(project(ck)) == render(project(m(ck)))
55+
for every request R and every mutation m over the harness-only flags of R's messages:
56+
render(project(R)) == render(project(m(R)))
5457
```
5558

5659
Equivalently: `project()` MUST be a pure function of the non-flag fields.
57-
Mutating `synthetic`, `harness_id`, or `ordinal` MUST NEVER change rendered
58-
bytes. If it does, a field was mis-bucketed as a flag when it is actually
60+
Mutating `synthetic`, `harness_id`, or `ordinal` on any message MUST NEVER change
61+
rendered bytes. If it does, a field was mis-bucketed as a flag when it is actually
5962
content. This is the CK analog of the render golden vectors and SHALL ship as a
60-
property test (§10).
63+
property test at request granularity (§10).
64+
65+
### 2.1 Request-level fields (NOT CK Message content)
66+
67+
CK Message is the **messages array only**. A provider REQUEST additionally carries
68+
render-config inputs that are NOT CK Message fields:
69+
70+
- **tool declarations** (the tool name + schema set) and **model** are
71+
**render-config EPOCH inputs** - HARD-bust markers (`tool_set_id`, `model_key`
72+
in the cache core), pinned per session-epoch exactly like the system block
73+
(§5.11). They are NOT carried on any `CkMessage`.
74+
- the **system block** is represented as pinned head `Role::System` message(s) for
75+
faithful round-trip (§5.11), and its bytes also feed the system HARD-bust marker.
76+
77+
So a request = `project(Vec<CkMessage>)` + `{ system, tools, model }` render-config.
78+
A change to any of the three render-config markers is a HARD bust (one unit, per
79+
the cache core's render-config epoch); the messages array byte-stability is what
80+
this spec governs. Response-only metadata (finish/stop reason, usage) is neither a
81+
CK Message field nor a request render-config input and is out of scope.
6182

6283
## 3. The CK type (normative Rust shape)
6384

‎docs/specs/mc-subc-module.md‎

Lines changed: 292 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,292 @@
1+
# MC SUBC module internals - normative specification
2+
3+
Status: **NORMATIVE (spec #5).** The transform brain: the SUBC daemon module that
4+
runs the harness-agnostic MC Transform over CK Message, consumes the cache-policy
5+
core, owns the frozen-set + durable store, and reconstructs byte-identically on
6+
restart. Companion specs: #1 CK Message (`ck-message.md`, the canonical it
7+
operates on), #2 codec (llm-runner provider-wire WireFamily), #3 MITM module, #4
8+
plugin<->subc connection (the wire protocol this module serves), indexed by
9+
`subconscious/docs/mc-subc-and-cache-foundations.md`.
10+
11+
Keywords MUST / MUST NOT / SHALL / SHOULD / MAY per RFC 2119. This spec defines
12+
the module's INTERNALS and the contract it presents to #4; it does NOT define the
13+
#4 transport, the #3 MITM interception, or the #2 provider-wire codecs (it
14+
references their boundaries).
15+
16+
---
17+
18+
## 1. What the module is
19+
20+
The MC module is a daemon-supervised, in-process-per-session **transform brain**.
21+
For each LLM round-trip a harness (or MITM shim) is about to make, the module:
22+
23+
1. receives the round-trip's message array as CK (decoded at the appropriate edge),
24+
2. classifies the pass (SOFT+ / SOFT / HARD) via the cache-policy core,
25+
3. on a bust pass: runs the reduction/compaction logic, renders byte-complete
26+
frozen units, and stores them atomically,
27+
4. on a defer pass: replays the frozen units verbatim (no render),
28+
5. returns the transformed CK array.
29+
30+
It is the OBSERVER consumer of the cache-policy core (it infers busts from the
31+
wire; it does NOT author render-config changes - those belong to the harness/user,
32+
or to llm-runner's author adapter on the owned path). The module owns Layer-2
33+
(render/store) for the MC harness family; the core owns Layer-0 (classify/coordinate).
34+
35+
**Boundary discipline (NORMATIVE):** the module and the cache core MUST contain NO
36+
provider branch (all provider knowledge is the downstream quirk pass / codec, spec
37+
#1 §1, #2) and the module MUST consume the cache core as an **in-process library**,
38+
never a per-pass RPC (per cache-core design: a per-step synchronous call to a
39+
separate service would add latency to every step and hang the loop if it stalls).
40+
41+
## 2. The transform contract (what #4 calls)
42+
43+
The module exposes ONE per-pass entry the connection layer (#4) drives:
44+
45+
```
46+
transform(session_id, pass_input) -> pass_output
47+
pass_input = {
48+
ck_messages, // the round-trip array as CK (§3 on how it arrives)
49+
input_identity, // opaque prefix fingerprint (cache-core §; anchor check)
50+
serializer_profile, // required (#1 §6); selects healing residual + quirk pass
51+
render_config, // { system_hash, tool_set_id, model_key } (cache-core)
52+
usage, // current context usage for scheduler/overflow
53+
}
54+
pass_output = {
55+
ck_messages, // transformed CK array to render to the wire
56+
action, // SOFT+ | SOFT | HARD (telemetry + #4 cache expectations)
57+
diagnostics, // optional: decision log row (off hot path)
58+
}
59+
```
60+
61+
- `transform` MUST be **pure with respect to the wire**: identical
62+
`(ck_messages, input_identity, render_config)` on a defer pass MUST yield
63+
byte-identical `pass_output.ck_messages` (the golden-vector SOFT+ assert).
64+
- `transform` MUST NOT block on network or another session's lock beyond the
65+
per-session actor (§7). It MAY perform local store I/O.
66+
- The module receives CK, not harness bytes: decoding happens at the edge (#3
67+
provider-wire codec on the MITM leg; the plugin shim's harness-model codec on
68+
the plugin leg - §8). The module is harness-agnostic by construction.
69+
70+
## 3. How the array arrives: always-full baseline + optional delta
71+
72+
Per Edge A (settled): **ALWAYS-FULL is the baseline.** The MITM leg has no delta
73+
channel (the module receives the full intercepted provider request as CK), so the
74+
always-full path is non-optional and MUST be the primary implementation.
75+
76+
- **Always-full:** `pass_input.ck_messages` is the complete post-compaction-marker
77+
array (plugin leg) or the full intercepted request (MITM leg). The module holds
78+
no cross-pass canonical content (Edge B); it reconstructs working state from this
79+
array + the durable frozen-set each pass. This makes the module **stateless in
80+
message CONTENT** and stateful only in cache-DECISION state.
81+
- **Plugin delta (OPTIONAL, post-measurement):** a future plugin-path optimization
82+
MAY send a tail-delta + fingerprint and have the module hold canonical in memory.
83+
This is gated on the ~550msg/~1MB round-trip measurement and MUST NOT be built
84+
before always-full. When built, a `NEED_FULL_SYNC` reply forces a full resend on
85+
cold/revert/first-pass. The always-full path remains the correctness baseline and
86+
the restart-recovery path regardless.
87+
88+
## 4. Consuming the cache-policy core (observer adapter)
89+
90+
The module drives the cache core's pure per-pass function (cache-core design,
91+
Round 5 frozen contract):
92+
93+
```
94+
core(prev_state, { signal, input_identity }) -> (new_state, action)
95+
state = { version, anchor_fingerprint, frozen_units:[{key,kind,frozen_payload}], pending_changes? }
96+
```
97+
98+
**4.1 Anchor check first (Leak C).** The core's first step compares
99+
`pass_input.input_identity` against `prev_state.anchor_fingerprint` over the
100+
covered prefix. The module MUST supply `input_identity` as a CONTENT fingerprint
101+
over the covered prefix (MC's `computeRawRangeFingerprint` shape: hash
102+
`ordinal:id:parts.length:partContentFingerprint`, raw content only, NEVER
103+
tag/drop/strip state). Match -> replay; diverge-within-coverage -> bust (discard
104+
frozen set, fresh render). Revert is host-caused = OBSERVED, so the anchor check
105+
is universal (the module is the observer adapter; this is its native mode).
106+
107+
**4.2 Signal mapping.** The module emits observer signals as objects `{kind, ...}`
108+
(never bare strings) from the first-cut kinds: `growing-tail | watermark-crossed-image
109+
| skeleton-window-moved | memory-delta | compartment-published | hard-fold-trigger
110+
| provider-nonce-only | revert-or-truncate | idle-ttl-expired`. The module does
111+
NOT populate `pending_changes` / author-policy fields (observer has no authored
112+
render-config lever).
113+
114+
**4.3 Classification consumes, never re-implements.** The module MUST NOT contain
115+
its own SOFT+/SOFT/HARD ladder; it maps its inferred conditions to core signals
116+
and consumes `action`. The HARD triggers MC owns (system-hash change, model/provider
117+
change, idle>TTL, project-memory epoch, structural compartment mutation, pressure
118+
backstop) are emitted as `hard-fold-trigger` signals with cause; the core classifies.
119+
120+
## 5. Frozen-set render/store layer (Layer 2 - the module owns this)
121+
122+
On a **bust** pass the module renders byte-complete frozen units; on a **defer**
123+
pass it places them verbatim. Render runs ONLY on bust passes (structural
124+
defer-safety - the bug-class kill).
125+
126+
**5.1 Frozen units are byte-complete payloads, not enum tags (Leak A).** Each unit
127+
is `{key, kind, frozen_payload}` where `frozen_payload` is the final rendered bytes
128+
(or complete deterministic inputs), produced by the module's renderer at freeze
129+
time. `kind in {drop | strip | skeleton | edit_marker | synthesized_region |
130+
injection}`. The module MUST NOT store `{id, enum}` and re-derive bytes on defer.
131+
132+
- **drop / strip:** the `[dropped §N§]` placeholder (a pure function of tag id,
133+
#1 §6.2) or empty-content shape per `decide_drop_shape(serializer_profile)`.
134+
- **skeleton / edit_marker:** the newest-N tool-skeleton (`tool_use` kept, output
135+
replaced) and superseded-edit compression (filePath + region-hint prefix) -
136+
rendered to final bytes at freeze, frozen as those bytes (the V4 Leak-A case).
137+
- **synthesized_region:** the full m[0] / m[1] block bytes (compartment history).
138+
Frozen as complete block bytes; on defer the module re-splices the frozen block
139+
against the freshly-rebuilt array at the frozen boundary id (a TRIM, not a
140+
content re-render - MC `inject-compartments.ts` reference behavior).
141+
- **injection:** deterministic synthetic-todowrite (`mc_synthetic_todo_<hash>`),
142+
frozen as its byte-complete tool_use/result pair.
143+
144+
**5.2 Atomic write-back (Leak B + D).** `new_state` is ONE value (units + markers +
145+
manifest), version-stamped (`new_state.version = prev.version + 1`), written by a
146+
**version-stamped CAS** (compare prev-version, swap, idempotent under retry). The
147+
module MUST NOT persist units, markers, or manifest separately (MC tore the cache
148+
when m[1] persisted but markers didn't). Single-writer (owned/MITM daemon) wins
149+
uncontended; the multi-writer case (shared store) retries - same core.
150+
151+
**5.3 The module never interprets `frozen_payload`.** The core freezes and replays
152+
opaque bytes; the render semantics live entirely in the module's renderer. This
153+
keeps the core harness-neutral.
154+
155+
## 6. Durable state + restart recovery (Edge B)
156+
157+
**6.1 What persists (cache-DECISION state, NOT message content).** The module's
158+
durable store (`store.db`, the MC harness's Layer-2 store, successor to `context.db`)
159+
holds:
160+
- the cache-core frozen-set state `{version, anchor_fingerprint, frozen_units}` per session,
161+
- the compartment history (m[0]/m[1] source: compartments + tiers + decay inputs),
162+
- the SURVIVING logic state (§9): scheduler/boundary, overflow/detected-limit,
163+
historian-failure, emergency-drain latch, note/auto-search state, usage,
164+
- the tags table (tag identity + cached token counts),
165+
- project memories + embeddings (unchanged subsystem).
166+
167+
Message CONTENT is NOT durably held by the module - it reconstructs from the
168+
harness full-array hand-off (the harness store remains the source of truth for
169+
message bytes; Edge B).
170+
171+
**6.2 Restart recovery (NORMATIVE, byte-identical).** On daemon restart the
172+
in-memory per-session actor is empty. The next pass arrives always-full (§3), so:
173+
1. the module rebuilds working state from the full CK array,
174+
2. re-applies the durably-persisted frozen-set against it (anchor check first: the
175+
restored `anchor_fingerprint` vs the live `input_identity`),
176+
3. on anchor match -> replays frozen units verbatim -> **byte-identical output to
177+
pre-restart**; on anchor mismatch (the session was reverted while the daemon was
178+
down) -> bust + fresh render (correct, not a leak).
179+
180+
Restart recovery MUST therefore be exactly `(full array) + (durable frozen-set) ->
181+
replay`, with no separately-reconstructed content. This is the §8 always-full path
182+
reused, which is why always-full is the non-optional baseline.
183+
184+
## 7. Per-session execution actor (Edge F)
185+
186+
The module MUST serialize all passes for a given `session_id` through a single
187+
in-process actor/lock (the subc coordinator precedent). Rationale: rapid
188+
consecutive passes for one session mutate one session's frozen-set; without
189+
serialization two passes race the CAS. Single-writer-per-session makes the §5.2
190+
CAS uncontended in the common case (the cross-process retry path remains for the
191+
shared-store / multi-daemon case). Distinct sessions run concurrently.
192+
193+
The transform is async at the #4 boundary (await-subc); the per-session actor
194+
bounds concurrency without cross-session leases (the lease subsystem COLLAPSES per
195+
the migration map - in-process serialization replaces cross-process leases).
196+
197+
## 8. The harness-model codec boundary (this module owns it)
198+
199+
Per the #1 §5.9 contract and the codec split: the **harness-model codecs**
200+
(`MessageV2 <-> CK`, `AgentMessage <-> CK`) are owned by THIS module (the plugin
201+
leg); the **provider-wire codecs** (`Anthropic-wire <-> CK`, `OpenAI-Responses <->
202+
CK`) are llm-runner's (#2, the MITM leg). The module:
203+
- on the PLUGIN leg: decodes the harness array (MessageV2 / AgentMessage) -> CK at
204+
ingress, transforms, encodes CK -> harness array at egress (the harness serializer
205+
then assembles + renders the wire; the module's quirk pass is gap-fill only, #1 §6.4).
206+
- on the MITM leg: receives CK already decoded by #2's provider-wire codec; the
207+
module transforms and returns CK; #2 re-renders + #3 forwards.
208+
209+
The harness-model codecs MUST satisfy #1 §5.9 ownership modes (identity-preserving
210+
`tool_call_id`, native-preserved signatures, pinned system block). The exact
211+
MessageV2/AgentMessage field mapping is specified alongside this module (a codec
212+
sub-spec); it is coordinated with llm-runner once #2 firms up, binding to #1 §5.9.
213+
214+
## 9. What relocates INTO the module vs into llm-runner vs SURVIVES
215+
216+
**9.1 Relocates INTO llm-runner (LLM-executing subagents leave the host).** The
217+
historian and dreamer are LLM-calling background workers. On the owned/SUBC path
218+
they run as **llm-runner sessions**, not opencode/pi host subagents. The MC module
219+
ORCHESTRATES them (decides when the historian fires from the in-memory tail trigger;
220+
schedules dreamer tasks via cron + conflict-domain leases) but the LLM execution is
221+
llm-runner's. The module MUST treat historian/dreamer as async producers whose
222+
writes (compartments, memories) ride the next natural bust and NEVER force a
223+
prompt-cache materialization (the background-task invariant survives verbatim).
224+
225+
**9.2 SURVIVES in the module (genuine session logic, per the migration map).**
226+
- the reduction/reclaim DECISIONS + the scheduler (execute/defer);
227+
- the protected-tail boundary (true-raw token sizing, open-arc fencing, live-prompt floor);
228+
- overflow detection + detected-limit persistence + the large-to-small model-switch arm;
229+
- the note system, auto-search hints, ctx_reduce nudge cooldown bands;
230+
- usage tracking; the tags table (identity + cached token counts);
231+
- the compartment trigger + decay renderer (the deterministic history manager);
232+
- the `[dropped §N§]` placeholder rendering.
233+
234+
**9.3 COLLAPSES (does not move - gone).** Empty-text sentinels (no array-index
235+
stability needed when the module holds post-strip state on always-full), the
236+
sentinel-vs-splice drop-mechanism split (one CK drop), the CAS/delta multi-writer
237+
DEFENSE for the single-daemon case (the CAS PRIMITIVE survives for the shared-store
238+
case, §5.2), the per-harness PARITY.md duplication (one transform + codecs).
239+
240+
**9.4 RELOCATES off the hot path.** The `§N§` re-prefix (incremental on the held
241+
array under the delta optimization; full-scan only on always-full bust), the
242+
per-strip watermark tables (unify into the ONE frozen-set, §5), the lease subsystem
243+
(in-process per-session actor, §7).
244+
245+
## 10. Daemon-down / failure (Edge D)
246+
247+
Fail-open-raw-passthrough is REJECTED (it double-busts or overflows - the D19.d/e
248+
finding). The #4 connection layer owns the plugin-side behavior (cache
249+
last-known-good per session; re-emit if the current anchor matches the
250+
last-known-good coverage; else abort the turn via the clean Error frame). The
251+
module's responsibility: on an internal transform failure it MUST return a clean
252+
Error result to #4 (never a partial/raw array), and MUST leave the durable
253+
frozen-set UNCHANGED (no half-written state) so the next pass either replays the
254+
last good frozen-set or busts cleanly. The CAS write-back (§5.2) gives this
255+
atomicity: a failed pass simply does not advance `version`.
256+
257+
## 11. Conformance checklist (a conforming module MUST)
258+
259+
1. Expose `transform(session_id, pass_input) -> pass_output` (§2), wire-pure on
260+
defer (byte-identical SOFT+ output).
261+
2. Consume the cache-policy core as an in-process library (§1, §4); contain NO
262+
SOFT+/SOFT/HARD ladder of its own and NO provider branch.
263+
3. Supply `input_identity` as a content fingerprint over the covered prefix
264+
(anchor check first, §4.1); classify revert/truncate as a bust, never SOFT+.
265+
4. Render byte-complete frozen units on bust only; replay verbatim on defer; never
266+
re-derive payload from moving state (§5.1, Leak A / V4).
267+
5. Persist the frozen-set as ONE atomic version-stamped CAS value (§5.2, Leak B/D).
268+
6. Persist only cache-DECISION state, never message content; reconstruct content
269+
from the always-full hand-off (§6.1).
270+
7. Recover byte-identically on restart = `(full array) + (durable frozen-set) ->
271+
anchor-checked replay` (§6.2); bust on anchor mismatch.
272+
8. Serialize passes per session through one actor; run distinct sessions
273+
concurrently; no cross-process leases for the single-daemon case (§7).
274+
9. Own the harness-model codecs (MessageV2/AgentMessage <-> CK) binding to #1 §5.9;
275+
gap-fill only on the plugin leg, never assembly (§8, #1 §6.4).
276+
10. Orchestrate (not host) historian/dreamer as llm-runner sessions; their writes
277+
ride the next bust and never force materialization (§9.1).
278+
11. On failure return a clean Error and leave the durable frozen-set unchanged
279+
(§10); never raw-passthrough.
280+
281+
## 12. Provenance
282+
283+
Consumes the cache-policy core contract (`subconscious/docs/cache-policy-core-design.md`,
284+
Round 5 frozen: per-pass function, `{version, anchor_fingerprint, frozen_units,
285+
pending_changes}` state, the 8 mechanics golden vectors V1-V8 with V4/V8 the
286+
leak-catches) and CK Message (`ck-message.md`). Surviving/collapsing/relocating
287+
logic per `docs/cache-policy/mc-subc-migration-map.md`. Edges B (durability), D
288+
(daemon-down), E (injection/delivery), F (per-session actor), G (compaction-marker
289+
plugin-only) and the always-full-baseline (Edge A) per
290+
`subconscious/docs/mc-subc-and-cache-foundations.md` §5-§6. The surviving
291+
subsystems (protected-tail, overflow, decay renderer, trigger, tags, notes,
292+
nudges) are MC's shipped implementations referenced as the behavior to preserve.

0 commit comments

Comments
 (0)