Skip to content

mental_model.md

Latest commit

 

History

History
732 lines (554 loc) · 31 KB

mental_model.md

File metadata and controls

732 lines (554 loc) · 31 KB
Raw

ShieldCraft Engine — Comprehensive Mental Model

Generated: 3 March 2026
Scope: Full repository scan — structure, architecture, data flow, subsystems, governance, and operational contracts.

1. What ShieldCraft Engine Is

ShieldCraft Engine is a spec-driven, deterministic code generation and governance engine written in Python 3.9+. It takes a product specification (the "spec") as input and produces:

  • A checklist of actionable items derived from the spec.
  • Generated code from templates (multi-framework support).
  • Evidence bundles proving lineage from spec → checklist → code.
  • Governance artifacts (manifests, provenance, refusal reports, readiness traces).

The engine's defining characteristics are:

  • Determinism: Identical inputs and repo state always produce identical outputs. No randomness, no timestamps in canonical artifacts, no network calls in core paths.
  • Auditability: Every decision is traceable — from spec pointer to checklist item to generated code — via lineage IDs, provenance headers, and evidence bundles.
  • Governance-first: A layered system of gates (G1–G22), contracts, and invariant assertions governs what the engine can and cannot emit. The engine will refuse to produce artifacts rather than produce incorrect or unverifiable ones.

2. Repository Structure

Top-Level Layout

src/shieldcraft/         # Core engine source code
tests/                   # Comprehensive test suite (~100+ test files)
docs/                    # Governance contracts, architecture docs, decision logs
  governance/            # Authoritative contracts (CHECKLIST_EMISSION, REFUSAL_AUTHORITY, etc.)
  persona/               # Persona protocol documentation
scripts/                 # CI helpers, audit tools, spec trial runners
templates/               # Multi-framework code generation templates
  frontend/              # Angular, Next.js, Svelte, Vue
  backend/               # Express, FastAPI, Flask
  api/, database/, infrastructure/, testing/, mobile/, common/
spec/                    # Canonical spec files and schemas
artifacts/               # Persisted run artifacts (execution state, snapshots, sync state)
.github/workflows/       # CI pipeline (spec-pipeline.yml)
products/                # Per-product output directories (plan.json, manifest.json, last_spec.json)
.selfhost_outputs/       # Self-host pipeline output directory

Key Configuration Files

File Purpose
pyproject.toml Poetry project config. Dependencies: jsonschema. Python ^3.9.
pytest.ini Test discovery: tests/ directory, test_*.py pattern.
copilot-instructions.md AI assistant instructions for working with the repo.
RELEASE_MANIFEST.json Release metadata.
repo_state_sync.json Repo sync state for deterministic verification.

3. Entry Points and Execution Modes

CLI Entry Point: src/shieldcraft/main.py

The CLI supports four modes:

  1. --validate-spec <SPEC_FILE> — Run preflight validation only {schema + governance + sync checks}.
  2. --self-host <SPEC_FILE> — Full self-host pipeline: ingest → validate → AST → checklist → codegen → emit artifacts to .selfhost_outputs/.
  3. --spec <SPEC_FILE> (default) — Standard run: load spec → validate → build checklist → optionally generate code/evidence.
  4. --spec <SPEC_FILE> --generate — Standard run + code generation.
  5. --spec <SPEC_FILE> --evidence — Standard run + evidence bundle generation.
  6. --spec <SPEC_FILE> --all — Full execute: plan + checklist + codegen + evidence + lineage + stability comparison.

Additional flags:

  • --dry-run — Preview without writing files.
  • --emit-preview <PATH> — Write preview JSON (with --dry-run).
  • --enable-persona — Enable persona evaluation (opt-in).
  • --schema <PATH> — Custom schema path (default: src/shieldcraft/dsl/schema/se_dsl.schema.json).

Programmatic Entry Points

  • Engine(schema_path) — Core engine class. Methods: run(), execute(), generate_code(), run_self_host(), run_self_build(), preflight(), generate_evidence().
  • engine_batch.run_batch(spec_paths, schema_path) — Batch processing of multiple specs.
  • api_server.py — Flask-based REST API (demo/prototype).

4. Core Pipeline — Data Flow

4.1 Spec Ingestion (services/spec/ingestion.py)

Raw file (JSON/YAML/TOML/text) → ingest_spec(path) → normalized dict
  • Reads raw bytes, tries JSON → YAML → TOML → fallback to raw text.
  • Non-dict results are wrapped in a minimal DSL skeleton via build_minimal_dsl_skeleton().
  • Dict results with model + sections keys are returned as-is (already DSL-shaped).
  • adapt_sections() normalizes dict-shaped sections into array format.

4.2 DSL Loading (dsl/loader.py)

ingest_spec(path) → optional canonical version enforcement → SpecModel or raw dict
  • Enforces dsl_version == "canonical_v1_frozen" when present.
  • Legacy specs (no version) are accepted with a deprecation warning.
  • Returns SpecModel(raw, ast, fingerprint) for canonical specs.

4.3 Preflight Validation (engine.preflight())

Sequential gate checks with short-circuit on failure:

  1. G2: Governance Presence Check — Verifies governance files exist in repo (services/governance/registry.py).
  2. G3: Repo Sync Verification — Verifies repo_state_sync.json integrity and artifact hashes (services/sync/__init__.py).
  3. G4: Schema Validation — Validates spec against JSON Schema (services/spec/schema_validator.py).
  4. G5: Spec Instruction Validation — _validate_spec() checks instruction sections, raises ValidationError.
  5. G6: Verification Spine — Asserts verification properties via verification/assertions.py.
  6. G7: Persona Veto Check — Records persona vetoes as advisory DIAGNOSTIC events (non-blocking under Phase 15 lock).
  7. G8: Test Attachment Contract — Verifies tests are attached when enforced.
  8. Snapshot Validation (optional) — When SHIELDCRAFT_SNAPSHOT_ENABLED=1, validates artifacts/repo_snapshot.json.

Each gate records events via ChecklistContext.record_event() for deterministic audit trail.

4.4 AST Construction (services/ast/builder.py)

spec dict → ASTBuilder().build(spec) → Node tree with pointers
  • Recursively builds a tree of Node objects from the spec dict.
  • Keys are sorted for determinism. Each node gets a JSON Pointer (ptr).
  • lineage_id computed per node. pointer_map built for lookups.
  • Node types: root, dict_entry, array_item, scalar.

4.5 Checklist Generation (services/checklist/generator.py)

The checklist generator is the compiler — transforms spec + AST into a finalized checklist. It operates in five phases (per COMPILATION_PHASE_MODEL.md):

Phase 1 — Ingestion: Extract raw items from spec sections. Phase 2 — Normalization: Classify items, compute severity, attach metadata, assign stable IDs. Phase 3 — Constraint Propagation: Apply spec constraints, dependencies, cross-section checks. Phase 4 — Synthesis: Merge derived tasks, invariant results, order ranks. Deduplicate and collapse equivalent items. Phase 5 — Finalization: Build evidence bundles, write manifests, compute diffs, validate cross-item constraints.

Key sub-modules used during compilation:

  • extractor.py — Extract items from spec sections.
  • classify.py / classifier.py — Classify items by type/category.
  • severity.py — Compute severity scores.
  • idgen.py — Generate stable, deterministic checklist item IDs.
  • dedupe.py — Deduplicate items.
  • collapse.py — Collapse equivalent items.
  • canonical.py — Canonical sorting.
  • deps.py / deps_tasks.py / implicit_deps.py — Dependency extraction and tasks.
  • cross.py — Cross-section validation checks.
  • flow.py — Compute execution flow and flow tasks.
  • order.py — Ordering constraints and rank assignment.
  • evidence.py — Build evidence bundles per item.
  • tier_enforcement.py — Enforce section tiers (A/B/C) and synthesize missing defaults.
  • quality.py — Compute checklist quality score.
  • outcome.py — Authoritative primary outcome derivation (REFUSAL > BLOCKED > ACTION > DIAGNOSTIC).

4.6 Code Generation (services/codegen/)

checklist items → FilePlan → TemplateEngine → rendered files with lineage headers
  • FilePlan maps checklist items to template files.
  • TemplateEngine renders templates (Jinja2-style) with spec context.
  • CodeGenerator.run() produces file outputs with injected lineage headers (engine version, lineage ID, source pointer).
  • Bootstrap items get special module_bootstrap.j2 templates.

4.7 Checklist Finalization (engine.finalize_checklist())

A guaranteed function that produces a canonical checklist result from any run state:

  • Collects all recorded gate events from ChecklistContext.
  • Derives primary_outcome using authoritative precedence (REFUSAL > BLOCKED > ACTION > DIAGNOSTIC).
  • Assigns roles to items: PRIMARY_CAUSE, CONTRIBUTING_BLOCKER, SECONDARY_DIAGNOSTIC, INFORMATIONAL.
  • Asserts semantic invariants (exactly one PRIMARY_CAUSE for non-ACTION outcomes, etc.).
  • Computes checklist quality score.
  • Attaches persona summary if persona events exist.
  • Never raises — always returns a result dict.

4.8 Self-Host Pipeline (engine.run_self_host())

The self-host mode is the engine running on its own spec to generate its own components:

  1. Validate spec.
  2. Build AST.
  3. Generate checklist.
  4. Filter bootstrap items.
  5. Run codegen on bootstrap items (dry-run).
  6. Optionally load and evaluate personas.
  7. Check selfhost input sandbox (allowed keys only).
  8. Check selfhost readiness marker.
  9. Check worktree cleanliness (unless SHIELDCRAFT_SELFBUILD_ALLOW_DIRTY=1).
  10. Write outputs to .selfhost_outputs/{fingerprint}/.
  11. Run minimality check (detect & collapse equivalent items; refuse on violations).
  12. Build execution plan (refuse on cycles, missing producers, priority violations).
  13. Generate repo snapshot.

4.9 Self-Build Pipeline (engine.run_self_build())

An extension of self-host that compares emitted artifacts against a baseline:

  • Runs self-host, copies outputs to artifacts/self_build/{fingerprint}/.
  • Compares bitwise against artifacts/self_build/baseline/v1/.
  • Detects mismatches and writes forensics.
  • Optionally establishes baseline when SHIELDCRAFT_SELFBUILD_ESTABLISH_BASELINE=1.

5. Governance and Gate System

5.1 Gate Identifiers (G1–G22)

Gates are deterministic checkpoints that record events as the engine progresses:

Gate Phase Purpose
G1 preflight Engine readiness failure
G2 preflight Governance presence check
G3 preflight Repo sync verification
G4 preflight Schema validation
G5 preflight Validation type gates (spec instruction validation)
G6 preflight Verification spine failure
G7 preflight Persona veto (advisory)
G8 preflight Test attachment contract
G9 compilation Generator fuzz gate
G10 compilation Generator prep missing
G12 generation Persona veto enforcement (legacy)
G13 generation Generation contract failed / Persona outcome override attempt
G14 post_generation Selfhost input sandbox / internal error return
G15 post_generation Disallowed selfhost artifact
G16 post_generation Minimality invariant failed
G17 post_generation Execution cycle detected
G18 post_generation Missing artifact producer
G19 post_generation Priority violation detected
G20 post_generation Quality gate failed
G22 generation Execute/codegen internal error return

5.2 Outcome Precedence (Authoritative)

REFUSAL > BLOCKED > ACTION > DIAGNOSTIC

Defined in services/checklist/outcome.py and enforced via derive_primary_outcome(). This is the single source of truth per CHECKLIST_OUTCOME_CONTRACT.md.

5.3 Refusal Authority System

Each REFUSAL gate maps to a named authority:

  • infrastructure — G1 (engine readiness), G3 (sync), G17 (cycles)
  • governance — G2 (governance presence), G5 (validation), G8 (test attachment), G16 (minimality), G18 (missing producer), G19 (priority)
  • persona — G7 (persona veto), G12 (persona enforcement)
  • selfhost — G14 (selfhost sandbox), G15 (disallowed artifact)
  • quality — G20 (quality gate)

Implemented in services/governance/refusal_authority.py.

5.4 Checklist Emission Contract

Every engine run must produce a persisted artifact — whether that's a checklist.json, refusal_report.json, checklist_draft.json, or errors.json. Silent termination without emitting an outcome artifact is a contract violation.

5.5 Authoritative Governance Documents

Located in docs/governance/:

Document Scope
CHECKLIST_EMISSION_CONTRACT.md Mandatory artifact emission
CHECKLIST_SEMANTICS.md Checklist item semantics
COMPILATION_PHASE_MODEL.md Fixed compiler phases
REFUSAL_AUTHORITY_CONTRACT.md Gate → authority mapping
AUTHORITY_CEILING_CONTRACT.md Authority limits
PERSONA_NON_AUTHORITY_CONTRACT.md Persona advisory-only boundary
TEMPLATE_NON_AUTHORITY_CONTRACT.md Templates non-authoritative
SPEC_TO_CHECKLIST_COMPILER.md Compiler contract
VERIFICATION_SPINE.md Verification responsibilities
TEST_ATTACHMENT_CONTRACT.md Test attachment enforcement
INFERENCE_EXPLAINABILITY_CONTRACT.md Inference traceability

6. Sync and Snapshot Subsystem

6.1 Repo Sync Verification (services/sync/__init__.py)

Ensures the repository state is known and verified before proceeding:

  • Reads repo_state_sync.json at repo root.
  • Verifies SHA256 of artifacts/repo_sync_state.json matches recorded hash.
  • Optional tree-hash staleness check.
  • Raises SyncError with frozen error codes: SYNC_MISSING, SYNC_HASH_MISMATCH, SYNC_INVALID_FORMAT, SYNC_TREE_MISMATCH.

6.2 Authoritative Sync Modes (verify_repo_state_authoritative())

Controlled by SHIELDCRAFT_SYNC_AUTHORITY env var:

Mode Behavior
repo_state_sync (default) Verify external sync file; missing = OK (non-mandatory).
external Deprecated. Requires SHIELDCRAFT_ALLOW_EXTERNAL_SYNC=1.
snapshot Use internal snapshot validation.
snapshot_mandatory Snapshot required; missing = fatal.
compare Verify both external and snapshot; compare hashes for parity.

6.3 Internal Snapshot (snapshot/__init__.py)

  • generate_snapshot(repo_root) — Walk repo, compute SHA256 per file, compute aggregate tree_hash.
  • validate_snapshot(path, repo_root) — Compare saved manifest to current repo.
  • Excludes: .git, .venv, node_modules, artifacts, .selfhost_outputs, dist, build.

7. Persona Protocol

7.1 Purpose

Personas are bounded, auditable decision agents that provide advisory guidance within engine workflows. Under the Phase 15 Persona Non-Authority Lock, personas are strictly advisory — they cannot alter checklist semantics or trigger refusals.

7.2 Architecture

src/shieldcraft/persona/
├── __init__.py          # Persona dataclass, load_persona(), find/resolve/detect conflicts
├── contract.py          # Allowed actions: annotate, veto, suggest, rank
├── decision_record.py   # record_decision() — persists via observability
├── persona_evaluator.py # evaluate_personas() — apply constraints/vetoes to items
├── persona_registry.py  # In-memory persona registry (register, clear, find by phase)
├── routing.py           # Static phase → persona routing table
└── runtime.py           # PersonaRuntime — constrained evaluator

7.3 Key Rules

  • Authority Classes (metadata-only): DECISIVE > ADVISORY > ANNOTATIVE. These do not grant runtime authority.
  • Allowed Actions: annotate, veto, suggest, rank.
  • Forbidden Mutations: Cannot modify id, ptr, severity, refusal, outcome, generated, artifact.
  • Determinism: All persona outputs deterministic for same inputs. Sorted by name for evaluation order.
  • Preconditions: SHIELDCRAFT_PERSONA_ENABLED=1 required. Repo sync and clean worktree checked.
  • Event Emission: All decisions recorded as PersonaEvents with deterministic companion hashes.

7.4 Evaluation Flow

  1. find_persona_files() discovers persona.json / personas/*.json.
  2. resolve_persona_files() picks highest version.
  3. load_persona() validates (schema + preconditions) and returns Persona dataclass.
  4. evaluate_personas() iterates personas sorted by name, applies veto/constraint rules.
  5. Vetoes recorded via emit_veto() → PersonaEvent. Constraints collected (not applied in-place).
  6. Disallowed constraint mutations logged as G15_PERSONA_CONSTRAINT_DISALLOWED diagnostics.

8. Interpreter Subsystem

8.1 Text Interpreter (interpreter/__init__.py)

Converts free-text/prose specs into structured checklist items:

  • Splits text into sentences (handles bullets, numbered lists, headings, paragraphs).
  • Scans for obligation keywords: must, must not, requires, should, refuse, unsafe, etc.
  • Produces ChecklistItem dicts with: id (deterministic hash), claim, obligation, risk_if_false, confidence, evidence_ref.
  • Items with keywords get confidence: "medium"; others get "low".
  • Always returns non-empty list (synthesizes from first line if needed).

8.2 Requirement Extraction (interpretation/requirements.py)

Extracts structured requirements from raw spec text for sufficiency and coverage analysis.

9. Checklist Quality and Sufficiency

Quality Scoring (services/checklist/quality.py)

Computes a 0–100 quality score for the checklist based on:

  • Synthesized default item count.
  • Insufficiency markers.
  • Attached to checklist as meta.checklist_quality. Score < 60 triggers a diagnostic item.

Sufficiency (sufficiency/, requirements/sufficiency.py)

Evaluates whether extracted requirements are adequately covered by checklist items:

  • evaluate_sufficiency(requirements, coverage) — returns verdict with mandatory counts.
  • Persisted as sufficiency.json and checklist_sufficiency.json.

Coverage (requirements/coverage.py)

Computes requirement → checklist item mapping:

  • compute_coverage(requirements, items) — returns coverage data.
  • Used by minimality, completion, and sufficiency evaluators.

Completeness (requirements/completion.py)

  • evaluate_completeness(requirements, items) — produces completion percentage.
  • is_implementable(summary, requirements) — determines if spec is implementable.

10. Code Generation (Multi-Framework)

Template-Based Code Generation (services/codegen/)

  • TemplateEngine renders templates with spec-derived context.
  • FilePlan maps checklist items to output file paths and templates.
  • CodeGenerator orchestrates: build plan → render → inject lineage headers → write.
  • Templates live in src/shieldcraft/services/codegen/templates/ (internal) and templates/ (external multi-framework).

External Framework Generators (src/shieldcraft/generators/)

Standalone generators for specific frameworks using Mustache templates (via chevron):

Generator Framework
angular_generator.py Angular
express_generator.py Express.js
fastapi_generator.py FastAPI
flask_generator.py Flask
nextjs_generator.py Next.js
react_native_generator.py React Native
svelte_generator.py SvelteKit
vue_generator.py Vue.js

Each generator:

  • Loads a spec file and template config.
  • Renders Mustache templates with spec-derived context.
  • Writes output files to a configured output directory.

External Templates (templates/)

templates/
├── frontend/    # Angular, Next.js, Svelte, Vue
├── backend/     # Express, FastAPI, Flask
├── api/         # API layer templates
├── database/    # Database templates
├── infrastructure/  # Infra templates
├── testing/     # Test templates
├── mobile/      # Mobile templates
├── common/      # Shared templates
└── languages/   # Language-specific templates

11. Observability (observability/__init__.py)

Execution State

  • emit_state(engine, phase, gate, status) — Appends state entries to engine._execution_state_entries.
  • Persisted to artifacts/execution_state_v1.json.
  • Entries: {phase, gate, status, error_code}. No timestamps (deterministic).

Persona Events

  • emit_persona_event(engine, persona_id, capability, phase, payload_ref, severity) — Records persona events.
  • Validated against persona_event_v1.schema.json.
  • Persisted to artifacts/persona_events_v1.json with companion hash file (persona_events_v1.hash).

Persona Annotations

  • emit_persona_annotation(engine, persona_id, phase, message, severity) — Records annotations.
  • Persisted to artifacts/persona_annotations_v1.json.

12. Verification Subsystem (verification/)

A comprehensive verification framework:

Module Purpose
registry.py Global property registry
assertions.py Assert verification properties
properties.py Verification property definitions
seed_manager.py Deterministic seed generation for reproducible runs
readiness_evaluator.py Evaluate readiness gates
readiness_report.py Render readiness reports
completeness_gate.py Completeness verification
coverage.py Test coverage analysis
determinism_contract.py Determinism verification
spec_fuzzer.py Speculative spec fuzzing for ambiguity detection
replay_engine.py Replay engine for determinism testing
metamorphic/ Metamorphic testing infrastructure
baseline.py Baseline comparison

13. Agents (agents/)

Specialized agents for automated tasks:

Agent Purpose
documentation_agent.py Generate product and code docs from spec + artifacts
test_synthesis_agent.py Synthesize test cases from specs
verification_agent.py Run verification tasks

14. Test Suite

Structure

Tests are organized by subsystem under tests/:

tests/
├── conftest.py              # Global fixtures (clear persona registry, mock sync, worktree clean)
├── analysis/                # Semantic gradient analysis tests
├── ast/                     # AST builder tests
├── checklist/               # Checklist generator, minimality, completeness tests
├── checklist_sufficiency/   # Sufficiency evaluation tests
├── ci/                      # CI-specific tests (smoke, output contracts, progress baseline)
├── cli/                     # CLI tests (validate-spec, emit-preview)
├── codegen/                 # Code generation tests
├── compilers/               # Compiler hardening tests
├── engine/                  # Engine integration tests
├── governance/              # Governance contract tests
├── persona/                 # Persona protocol boundary tests
├── selfhost/                # Self-host pipeline tests
├── selfbuild/               # Self-build pipeline tests
├── snapshot/                # Snapshot verification tests
├── sync/                    # Sync verification tests
├── spec/                    # Spec validation, schema compliance tests
├── verification/            # Verification subsystem tests
├── verdict/                 # Verdict aggregation tests
├── release/                 # Release regression tests
└── (60+ top-level test files covering cross-cutting concerns)

Key Test Fixtures (conftest.py)

  • clear_persona_registry — Autouse: clears persona registry between tests.
  • default_external_sync — Autouse: sets SHIELDCRAFT_SYNC_AUTHORITY=external and SHIELDCRAFT_ALLOW_EXTERNAL_SYNC=1 for test isolation. Monkeypatches _is_worktree_clean to return True.

15. Environment Variables

Variable Purpose Default
SHIELDCRAFT_PERSONA_ENABLED Enable persona evaluation "0"
SHIELDCRAFT_SNAPSHOT_ENABLED Enable snapshot validation in preflight "0"
SHIELDCRAFT_SYNC_AUTHORITY Sync verification mode "repo_state_sync"
SHIELDCRAFT_ALLOW_EXTERNAL_SYNC Allow deprecated external sync "0"
SHIELDCRAFT_SELFBUILD_ALLOW_DIRTY Skip worktree cleanliness check "0"
SHIELDCRAFT_SELFBUILD_ENABLED Enable self-build mode "0"
SHIELDCRAFT_SELFBUILD_ESTABLISH_BASELINE Create baseline from current build "0"
SHIELDCRAFT_ENFORCE_TEST_ATTACHMENT Enforce test attachment contract "0"
SHIELDCRAFT_BUILD_DEPTH Self-build depth counter "0"

16. CI Pipeline (.github/workflows/spec-pipeline.yml)

  1. Checkout code.
  2. Setup Python 3.11.
  3. Install dependencies (pip install -e . pytest).
  4. Regenerate codegen outputs (dry-run self-host with SHIELDCRAFT_SELFBUILD_ALLOW_DIRTY=1).
  5. Clean stale bytecode.
  6. Run full test suite (pytest -q).
  7. Run self-host tests specifically.
  8. Upload test failure artifacts.

17. Key Architectural Patterns

17.1 Deterministic Canonicalization

Everything is canonicalized for determinism:

  • util/json_canonicalizer.py — canonicalize(obj) produces compact, sorted-key JSON.
  • Spec fingerprints are SHA256 of canonicalized spec.
  • File hashes use SHA256 everywhere.
  • IDs are deterministic hashes of (pointer + text).

17.2 Event-Sourced Gate System

The ChecklistContext acts as an event store:

  • Gates record events during execution.
  • finalize_checklist() derives outcomes from accumulated events.
  • This pattern ensures that any path through the engine produces a deterministic outcome.

17.3 Never-Fail Finalization

finalize_checklist() is designed to never raise. It catches all exceptions internally and produces a result dict regardless. This guarantees the emission contract.

17.4 Authority Ceiling

Strict hierarchy of what can influence outcomes:

  • Governance/engine contracts → can cause REFUSAL.
  • Personas → advisory only; cannot cause REFUSAL.
  • Templates → non-authoritative; cannot influence semantics.

17.5 Compilation Model

The checklist generator is conceptualized as a compiler:

  • Spec is the "source code".
  • AST is the intermediate representation.
  • Checklist items are the "compiled output".
  • The compiler has fixed phases, deterministic transforms, and invariant assertions.

18. Known Operational State

Current Blockers

  • Sync gate (sync_not_performed): The engine's preflight() calls verify_repo_state_authoritative(). In the default repo_state_sync mode, if the sync file is missing, it's non-mandatory and proceeds. But in the run_self_host() path, additional checks for _last_sync_verified attribute and _last_validated_spec_fp can prevent the pipeline from completing, resulting in a sync_not_performed RuntimeError that gets caught and emitted as a diagnostic in the refusal report.

  • Workaround: Setting SHIELDCRAFT_SELFBUILD_ALLOW_DIRTY=1 bypasses the worktree cleanliness check. The main.py self-host wrapper uses a pre-scan approach to produce checklist_draft.json before the engine pipeline runs, ensuring draft artifacts are always emitted even if the engine itself fails.

Worktree and Branch State

  • main branch is the primary branch.
  • remediation/remove-unintended-edits branch was created for safe remediation.
  • Local modifications may exist in engine.py, main.py, checklist/generator.py.

19. Data Model Reference

Spec (Input)

{
  "metadata": { "product_id": "...", "source_format": "..." },
  "model": { "modules": [...], "dependencies": [...] },
  "sections": [ { "id": "...", "items": [...] } ],
  "instructions": { ... },
  "invariants": [ ... ]
}

Checklist Item

{
  "id": "8-char-hash",
  "ptr": "/section/path",
  "text": "Implement X from /ptr",
  "severity": "high|medium|low",
  "classification": "compiler|bootstrap|...",
  "confidence": "high|medium|low",
  "priority": "P0|P1|P2",
  "blocking": true|false,
  "role": "PRIMARY_CAUSE|CONTRIBUTING_BLOCKER|SECONDARY_DIAGNOSTIC|INFORMATIONAL",
  "meta": {
    "gate": "G4_SCHEMA_VALIDATION",
    "phase": "preflight",
    "tier": "A|B|C",
    "source": "default|inferred|...",
    "inference_type": "heuristic|safe_default|...",
    "justification": "...",
    "justification_ptr": "/..."
  },
  "evidence": {
    "quote": "...",
    "source": { "ptr": "/...", "line": 42 },
    "source_excerpt_hash": "12-char-hash"
  }
}

Finalized Checklist Result

{
  "checklist": {
    "items": [...],
    "emitted": true,
    "events": [...],
    "primary_outcome": "REFUSAL|BLOCKED|ACTION|DIAGNOSTIC",
    "refusal": true|false,
    "refusal_reason": "...",
    "refusal_authority": "governance|persona|infrastructure|selfhost|quality",
    "blocking_reasons": [...],
    "confidence_level": "high|medium|low",
    "persona_summary": { ... },
    "meta": { "checklist_quality": 0-100 }
  },
  "primary_outcome": "...",
  "refusal": true|false,
  "emitted": true,
  "error": { "message": "...", "type": "RuntimeError" }
}

Self-Host Output

.selfhost_outputs/
├── {fingerprint}/
│   ├── bootstrap_manifest.json
│   ├── checklist.json
│   ├── repo_snapshot.json
│   └── modules/...
├── checklist_draft.json     (always emitted by main.py wrapper)
├── refusal_report.json      (when engine returns error/refusal)
├── summary.json             (validation failure path)
├── errors.json              (validation error path)
├── sufficiency.json         (best-effort)
└── requirements.json        (best-effort)

20. Glossary

Term Definition
Spec Product specification — JSON/YAML/TOML/text input describing what to build.
DSL Domain-Specific Language — the structured spec format (se_dsl_v1).
AST Abstract Syntax Tree — intermediate representation of the spec.
Checklist The compiled output — actionable items derived from the spec.
Gate A deterministic checkpoint (G1–G22) that records pass/fail events.
Preflight Validation phase before any work begins.
Self-host The engine generating its own components from its own spec.
Self-build Self-host + baseline comparison for bitwise determinism verification.
Persona An advisory decision agent that provides guidance without authority.
Finalization The guaranteed-success function that produces canonical checklist output.
Refusal An explicit, auditable decision not to emit artifacts because preconditions were unmet.
Evidence bundle Cryptographic proof of lineage from spec → checklist → code.
Sync verification Checking that repository artifacts match recorded hashes.
Snapshot Internal file manifest with SHA256 hashes for determinism verification.

End of mental model.