Skip to content

feat(ladder): plan-016 online promotion ladder — verifier-gated, per-tenant + system learning proven#2

Open
JayFarei wants to merge 9 commits into
mainfrom
worktree-ladder-016
Open

feat(ladder): plan-016 online promotion ladder — verifier-gated, per-tenant + system learning proven#2
JayFarei wants to merge 9 commits into
mainfrom
worktree-ladder-016

Conversation

@JayFarei

@JayFarei JayFarei commented Jul 2, 2026

Copy link
Copy Markdown
Owner

Closes #1

What this is

The plan-016 online promotion ladder, built end to end and gated by its own grounded verifier: a typed substrate (typed answer contracts, snapshot mount + drift fingerprints, measured-turn executor), a shadow-pair counterfactual harness (every pair runs exposed vs masked on the same query and snapshot, interleaved, with the masked arm's on-disk prompts provably free of library surface), and a verifier-gated promotion ladder (quarantine → shadow → candidate → promoted → deprecated) driven by pre-registered boundaries frozen in prereg/ladder-boundaries.json before the first episode existed.

Proof it works (re-runnable from this branch)

./verify/ladder.sh eval/ladder/runs/demo → GREEN on all 15 checks, exit 0 (born red at plan creation, f4e378eee9). The evidence is committed; any reviewer can re-run the verifier against it, and pnpm ladder:demo regenerates it at the pinned commit. 65/65 vitest units pass.

Key checks, each a frozen regression against a failure this repo shipped before:

  • V4:inert — the gate is live in both directions (2 promotions, 2 rejections), the inversion of the 0/22 inert-gate incident.
  • V4:drift — forced-drift probe observed on both edges: promoted → quarantine + abstention, with pre-drift evidence archived so a demoted procedure re-earns promotion on fresh pairs only.
  • V5 — every promotion is boundary-backed (≥30 pairs, win-floor 0.70); usage alone never promotes.
  • V6 — every promoted procedure is load-bearing: replay is anchored to the recorded snapshot fingerprint and recorded answer, and removal changes the replayed answer.
  • V7:G1 — post-promotion win-floor earned on query windows never seen before promotion (holdout asOf 21–24 vs promotion 9–19; 10 distinct query identities per promoted procedure, answers varying with the window).
  • V7:G2 — the second tenant (schema-distinct orders corpus) runs with a zero src/ diff and reaches a decisive terminal state.

Per-user and system learning, shown in artifacts

  • Per-user: alpha (support-tickets) promotes alpha-open-high-count + alpha-open-topics-index; beta (orders) promotes beta-delivered-sum + beta-open-regions-index. Divergent promoted sets from each tenant's own traffic; dashboard in eval/ladder/runs/demo/REPORT.md + curves.json (DL-per-intent, inline-rederivation curve, cumulative turns avoided).
  • System: alpha's winner is suggested into beta's quarantine and must earn promotion from beta's own paired evidence; on beta's schema it abstains, loses every pair, and beta's gate records the rejection (earn-or-stay-put, decline path live in the run; promote path covered by a unit test).

Honest boundary

The driver is a deterministic scripted policy over real mounted fixtures (driver:"scripted" on every artifact; synthetic clock disclosed in the REPORT). Every number is measured by the executor (turns are paged store reads, never constants) and every probe is actually run. This PR proves the ladder mechanics; it does not claim helpers beat inline for a frontier model, nor that agent-authored procedures promote under live traffic (plan 016 P4, explicitly open).

Review trail

Built via a staged multi-agent workflow: Opus build agents, Fable design gates re-observing each stage, Codex external code review (read-only sandbox). All confirmed findings fixed: replay anchored to recorded world-state (F1), demote resets live evidence (F2), prompt-hash row binding (F3), distinct-query holdout/genericity (F4, the load-bearing one), evidence regenerated at the commit containing its generator (F5), measured adversarial turns (F6), wrong-task seed guard (F7). Verifier and prereg are checksum-pinned and untouched since f4e378eee9 (0-line diff).

Follow-ups (out of scope, tracked in #1 discussion): extend verify/.checksums to pin fixtures and seeds; exercise the promote half of cross-tenant suggestion in a live run; P4 (df.author agent-authored candidates on live traffic).

🤖 Generated with Claude Code

https://claude.ai/code/session_01DsqpWFqTSQd9Py7gK4TqvF

JayFarei and others added 9 commits July 1, 2026 23:40
@JayFarei @claude
docs(kb): plan 016 online promotion ladder + born-red grounded verifi…
f4e378e 
…er + prereg boundaries

Plan 016 supersedes the 014/015 offline bake-off framing with an online
verifier-gated promotion ladder (loopcraft-forged). verify/ladder.sh is
checksum-pinned and born-red at HEAD; prereg/ladder-boundaries.json is
frozen before any episode so V0 ancestry holds. Includes plan 015 and the
structured-learning design report the review ran against.

Refs #1

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DsqpWFqTSQd9Py7gK4TqvF
@JayFarei @claude
feat(ladder): S1 core substrate
76adfdd 
Refs #1

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@JayFarei @claude
fix(ladder): S1 gate fixes
b1876e8 
Refs #1

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@JayFarei @claude
feat(ladder): S2 pairing + state machine + controls
433d262 
Refs #1

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@JayFarei @claude
feat(ladder): S3 demo + committed evidence
c46cfd5 
Refs #1

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@JayFarei @claude
fix(ladder): apply review findings
5808341 
- replay: anchor V2/V6 to recorded world-state (assert mounted fingerprint ==
  episode.snapshotHash and replay reproduces the committed answer before a
  replay/ablation counts as checked) so reproducibility, not just in-process
  determinism, is proven.
- stateMachine.demote: reset live bookkeeping and archive pre-drift evidence so a
  demoted procedure re-earns promotion on fresh pairs instead of instantly
  re-promoting on pre-drift wins.
- seed: abstain when a seed lands on a task other than its own (guards against a
  wrong-task serve being scored as a spurious win).
- demo: measure the adversarial prose-in-string row's turns via ExecContext (no
  hard-coded constant); reword the masked prompt to reference the snapshot by
  fingerprint + record count (drop the false "records below" claim); record a
  promptHash tying each row to its on-disk prompt and assert masked prompts carry
  zero library tokens at generation.
- REPORT: disclose the synthetic clock, the repeated-identical-traffic limit, the
  counterfactual nature of the inline-rederivation curve, and that only the
  decline half of cross-tenant earn-or-stay-put is exercised in the run.
- tests: cover demote-reset and the promote-on-own-evidence system-learning branch.

Refs #1

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DsqpWFqTSQd9Py7gK4TqvF
@JayFarei @claude
chore(ladder): regenerate demo evidence at the fix commit
fd251aa 
Re-stamps episodes.jsonl / REPORT / prompts with the commit that actually
contains the generator, closing the review finding that evidence was stamped
with a commit that could not reproduce it (commit generator first, run, commit
artifacts second). Verifier GREEN on all 15 checks.

Refs #1

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DsqpWFqTSQd9Py7gK4TqvF
@JayFarei @claude
fix(ladder): earn holdout and genericity on distinct seeded queries
6247dce 
Codex review F4: each procedure's pairs previously repeated one identical
task/query/answer, so the quarantine->shadow genericity rung and the V7:G1
post-promotion holdout passed on identical traffic (vacuous generalisation).

- TaskParam { asOf }: every pair instantiates its task on a distinct 'as of
  the first N records filed' window (distinct query text, distinct answers
  where the data supports it); records are filing-ordered so the window is a
  prefix and inline cost varies with it (measured via scanPrefix).
- Genericity now counts DISTINCT query identities (queryKey) — identical
  replays of one literal never advance quarantine->shadow; evidence records
  distinctQueries.
- Holdout windows are strictly later than every promotion window, so the
  post-promotion win-floor is earned on record windows never seen before
  promotion.
- Seeds become window-capable (position-aware indexes: match positions,
  first-occurrence maps, prefix sums) — one index serves the whole query
  family, which is what makes the seed win distinct queries rather than one
  memorised literal. Stale-clone control and probes updated likewise.
- Curves now cost each exposed episode against the masked row of the SAME
  pair; per-window prompt files tied to rows by promptHash.
- Tests: 65 pass, including new 'identical replays never leave quarantine'
  and distinct-traffic promote-on-own-evidence coverage.

Refs #1

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DsqpWFqTSQd9Py7gK4TqvF
@JayFarei @claude
chore(ladder): regenerate demo evidence at the distinct-query fix commit
5b781aa 
Evidence now stamps 6247dce (which contains its generator): 281+220
episodes over distinct seeded windows, 10 distinct queries per promoted
procedure (promotion windows 9-19, holdout 21-24 strictly unseen before
promotion), answers varying with the window. Verifier GREEN 15/15.

Refs #1

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DsqpWFqTSQd9Py7gK4TqvF
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Build online promotion ladder (plan 016): typed substrate, shadow pairs, verifier-gated promotion

1 participant

@JayFarei