Skip to content

ARCHITECTURE.md

Latest commit

 

History

History
115 lines (85 loc) · 7.58 KB

ARCHITECTURE.md

File metadata and controls

115 lines (85 loc) · 7.58 KB
Raw

Architecture

This document goes a level deeper than the README: the control flow, the state on disk, the agent contracts, and the reasoning behind the non-obvious choices.

Mental model

Flux is a hill-climbing search over the space of skill sets, where:

  • a state is the current set of skills for a domain (the "frontier"),
  • a move is "add or edit one skill,"
  • the objective function is the mean rubric score on a held-out validation batch, computed by an LLM judge,
  • and a move is accepted only if it strictly improves the objective.

Because there's no ground-truth objective, Flux constructs one per domain (the rubrics) and works hard to keep that constructed objective honest (the evaluator's anti-gaming rules).

Control flow

The /flux orchestrator drives everything. One evolve run:

  1. Initialize domain — derive a canonical domain ID (e.g. "React 19 frontend development" → react-19-frontend), register it, and create the on-disk state tree. Fuzzy-matches existing domains to avoid duplicates.
  2. Analyze domain (once) — spawn flux-domain-analyst to produce profile.json + challenges.json.
  3. Loop for N iterations:
    • 3a Select 3–5 unused training challenges (prefer diversity across sub-domains/difficulty).
    • 3b Execute — flux-executor attempts them with the current skills.
    • 3c Evaluate — flux-evaluator scores against rubrics.
    • 3d Collect failures — challenges scoring < threshold (default 0.7).
    • 3e Propose — flux-proposer clusters failures into ONE proposal, informed by proposal history + experience bank.
    • 3f Build — flux-builder materializes the candidate into staging/.
    • 3g Validate — re-run executor + evaluator on the held-out validation batch with the staged skill added.
    • 3h Promote or reject — if mean validation score beats the frontier, promote; else archive to rejected/.
    • 3i Persist — append to history.json, write the iteration log.
  4. Summarize — score progression, discovered/rejected skills, remaining gaps.

State is persisted after each iteration, which is what makes --resume possible.

State on disk

Per domain, under ~/.claude/flux/domains/{domain}/:

profile.json              # domain analysis: key concepts, sub-domains, version snapshot
challenges.json           # all challenges + rubrics + train/validation split
frontier.json             # current best: score, skills[], iteration, used_training_ids[]
history.json              # every proposal + its outcome (audit trail)
experience_bank.json      # accumulated {condition, action, rationale} insights
iterations/{i}/           # per-iteration artifacts:
    training_batch.json       #   challenges selected
    solutions.json            #   executor output
    evaluation.json           #   evaluator scores
    proposal.json             #   proposer output
    validation_*.json         #   held-out run
    log.json                  #   iteration metadata
staging/                  # candidate skill under validation (cleared each iteration)
rejected/                 # archived candidates that didn't beat the frontier

Promoted skills are installed to ~/.claude/skills/{domain}/{skill}/SKILL.md, where Claude Code discovers them globally.

Separating frontier (the accepted state), staging (the current candidate), and rejected (the archive) keeps the search auditable — you can see every move that was tried and why it was or wasn't accepted.

Agent contracts

Every agent has a strict JSON I/O contract so the orchestrator can chain them deterministically. The orchestrator always passes absolute file paths — agents never infer paths.

Domain Analyst

  • In: domain description, output dir.
  • Out: profile.json, challenges.json.
  • Notable rules: rubric weights must sum to 1.0; ≥2 "boundary-awareness" challenges per domain (when not to apply a pattern) to stop skills from becoming dogmatic; challenges must be solvable for any project matching the domain, not a specific toolchain; "build X" over "explain X."

Executor

  • In: challenges file, skills dir(s), output file.
  • Out: solutions.json with execution_trace, files_created, self_assessment, blockers.
  • Notable rules: must actually execute (write code, run commands) in a mktemp -d sandbox, not describe; must be honest — inflated self-assessments corrupt the evolution signal.

Evaluator (the quality signal)

  • In: solutions file, challenges file (rubrics), output file.
  • Out: evaluation.json with per-criterion scores + reasoning, failure modes, strengths, summary stats.
  • Notable rules: ultrathink on every scoring decision; reward correctness most; penalize untested claims; calibration anchors fix what 0.3 vs 0.7 vs 0.9 mean; "do not inflate scores."

Proposer

  • In: evaluation (failures), profile, experience bank, history summary, current skills, output file.
  • Out: proposal.json — exactly ONE intervention (new_skill | edit_skill | new_experience) with the failures it addresses.
  • Notable rules: one thing per iteration; highest-impact gap first; prefer editing a near-miss skill over adding a new one; don't re-propose something history shows was rejected.

Builder

  • In: proposal, profile, experience bank, staging dir, skills dir (read-only).
  • Out: staging/{skill}/SKILL.md + experience_bank_updates.json.
  • Notable rules: bake experiences into prose (no [exp-001] citation tags); under ~200 lines, state each point once; end every skill with a version_context block; verify technical claims against official docs via WebSearch/WebFetch.

Why an LLM judge that distrusts itself

The first naive version of the loop converged to ~0.93 within two iterations — and the skills read like test-prep. The model was writing solutions for the grader, and the grader was rewarding it. Two structural rules in the evaluator (and matching framing in the proposer/builder) fix this:

  1. Anti-evaluation-awareness: penalize solutions framed for a reviewer instead of a user.
  2. Anti-overclaiming: penalize absolute language ("always/never/must") used without caveats; prefer "prefer / default to" with explicit hard-rule vs. soft-suggestion tiers.

This is the single most important piece of the design: in a system with no ground truth, the integrity of the constructed objective is everything.

Failure modes found and fixed

Each of these was discovered by reading actual rejected candidate skills and tracing the pathology back to the responsible agent:

  1. Evaluator-oriented framing → rewrite for a senior engineer, not a grader.
  2. Overclaiming → "prefer / default to" language with normative tiers.
  3. Hallucinated APIs → give the Builder WebSearch/WebFetch to verify against docs.
  4. Fake [exp-XXX] citation tags polluting skill text → bake experiences into prose.
  5. Verbosity → target ~half the first-draft length.
  6. Domain Analyst assuming a sub-context → keep challenges domain-universal.
  7. Premature convergence on strong-baseline domains → --threshold 0.85 flag.
  8. Skills doing five things at once → single-concern, <200-line rule.
  9. Mixed hard/soft guidance → explicit normative layering with tier labels.

Known architectural limits

See the README's Honest Limitations. The biggest one: it's a specification executed interactively by Claude Code, not a standalone runtime, so it can't run unattended or in CI. The cleanest v2 is a thin Claude Agent SDK program that drives the same five agent contracts.