Skip to content

LEARNING_STRATEGY.md

Latest commit

 

History

History
303 lines (223 loc) · 10.4 KB

LEARNING_STRATEGY.md

File metadata and controls

303 lines (223 loc) · 10.4 KB
Raw

ShieldCraft Engine — Self-Learning Test Strategy

Status: Level 1 (project-level) implemented. Level 2 (feature-level) is Phase 2.

Overview

The ShieldCraft engine is fully deterministic: the same spec always produces the same checklist. "Self-learning" therefore does not mean the engine mutates its weights. It means the evaluation pipeline produces structured gap reports that precisely name which engine function produced a deficient output, and those gaps are rendered as (a) executable failing tests that define the improvement backlog, and (b) AI-ingestable prompts that ask an LLM to suggest concrete code changes.

The loop closes when a human (or an AI agent operating at the code level) applies one of those suggestions and re-runs the learning trial — at which point previously failing tests go green and the corpus score rises.

Architecture

spec_corpus
    │
    ▼
scripts/run_learning_trial.py           ← orchestrator
    │
    ├─► Engine.run_self_host()          ← existing engine (unchanged)
    │       └─► .selfhost_outputs/      ← pipeline artifacts
    │               checklist.json
    │               checklist_quality.json
    │               spec_coverage.json
    │               checklist_sufficiency.json
    │               manifest.json
    │               ...
    │
    ├─► sufficiency/evaluator.py        ← existing: compute complete_pct
    │
    ├─► learning/gap_analyzer.py        ← NEW: classify gaps by source module
    │       └─► gap_report.json
    │
    ├─► learning/code_suggestion_formatter.py  ← NEW: render prompts for AI
    │       └─► improvement_prompts.md
    │           ai_suggestions.json  (--ai-mode)
    │
    └─► learning/dynamic_test_generator.py     ← NEW: write executable pytest tests
            └─► tests/generated/test_learning_{iter}_{spec}.py

File Inventory

File Role
scripts/spec_synthesizer.py Generates the parametric spec corpus (32 fixed variants across 4 tiers) and random fuzz specs
scripts/run_learning_trial.py Main orchestrator — runs iterations, tracks history
scripts/run_fuzz_suite.py Fuzz evaluation driver — generate N random specs, run engine, report pass rate
src/shieldcraft/learning/__init__.py Package marker
src/shieldcraft/learning/gap_analyzer.py Reads pipeline output, classifies gaps, writes gap_report.json
src/shieldcraft/learning/code_suggestion_formatter.py Formats gaps as LLM prompts with source snippets
src/shieldcraft/learning/dynamic_test_generator.py Writes executable pytest tests from gap evidence
tests/analysis/test_semantic_gradient.py Extended with test_semantic_gradient_is_monotonic()
tests/property/test_fuzz_property.py 500 parametrised property tests, one per random seed

Spec Corpus (Tier Model)

The synthesizer creates specs at four maturity levels:

Tier Name Contains
0 raw Plain prose text, no DSL structure
1 structured metadata + sections + model skeleton
2 partial Tier 1 + invariants + instructions
3 full All 8 DSL keys, mirrors demos/valid.json

Within each tier, the synthesizer varies section_count (1, 3, 6), invariant_count (0, 2, 5), prose density (sparse/dense normative keywords), and presence of agents, artifact_contract, and pipeline — producing 32 distinct fixed spec variants total.

The synthesizer also supports unlimited random fuzz specs via random_spec(seed) (see Fuzz Suite below).

Generate the corpus with:

python scripts/spec_synthesizer.py --out spec_trials/generated

Running the Learning Loop

# Minimal — 3 iterations, auto-bypass sync gate
python scripts/run_learning_trial.py spec_trials/generated

# Explicit options
python scripts/run_learning_trial.py spec_trials/generated \
    --max-iterations 5 \
    --out learning_artifacts \
    --tests-dir tests/generated

# With active LLM suggestions
export SHIELDCRAFT_LLM_ENDPOINT=https://your-llm-api/completions
export SHIELDCRAFT_LLM_API_KEY=sk-...
python scripts/run_learning_trial.py spec_trials/generated --ai-mode

The runner automatically sets SHIELDCRAFT_SELFBUILD_ALLOW_DIRTY=1 to bypass the sync gate (required when running against this repo without a passing sync).

Convergence Criterion

A single spec converges when:

  • complete_pct >= 0.98 (from sufficiency/evaluator.py)
  • readiness grade is A or B (from guidance/readiness.py)

The corpus converges when all specs have converged.

The corpus_score composite (written to learning_artifacts/history.json) is:

total_score = complete_pct × 0.5  +  grade_numeric × 0.3  +  quality_ratio × 0.2

where grade_numeric maps A→1.0, B→0.75, C→0.5, F→0.0.

Gap Taxonomy

Gap type Failure signal Source module Source function
MANDATORY_UNCOVERED P0/P1 req is PARTIAL/UNBOUND requirements.coverage compute_coverage
MISSING_DIMENSION Req has uncovered dimensions requirements.completion evaluate_completeness
ORPHAN_SPEC_CLAUSE Spec pointer has no covering item services.checklist.generator _extract_from_ast
LOW_SIGNAL_ITEM Item flagged as low-signal checklist.quality evaluate_quality
INVALID_ACTIONABILITY Item failed imperative-verb check checklist.quality evaluate_quality
HEURISTIC_OVERRIDE Sufficiency granted by volume only sufficiency.evaluator evaluate_from_files

Generated Tests

Three categories of tests are written to tests/generated/:

Coverage tests (regression guards)

Assert that HIGH-confidence items from the engine's own output remain present in future runs. These start green and must stay green.

def test_coverage_item_abc123(engine_output_t3_s3_inv2_full):
    # asserts ptr '/section_1/authentication' is in checklist

Orphan elimination tests (improvement targets)

Assert that formerly orphaned spec clauses are now covered. These start failing — they define the engine improvement backlog.

def test_orphan_eliminated_section_2_auth(engine_output_t3_s3_inv2_full):
    # asserts '/section_2' is NOT in spec_coverage.json uncovered_units

Quality tests (improvement targets)

Assert that previously-invalid items now have quality_status == VALID. These also start failing.

def test_quality_valid_item_xyz(engine_output_t3_s3_inv2_full):
    # asserts item 'xyz' quality_status == 'VALID'

Run generated tests:

pytest tests/generated/ -v

Improvement Prompts

For each gap, the formatter reads the live source of the named engine function via inspect.getsourcelines() and composes a structured prompt:

## Gap: ORPHAN_SPEC_CLAUSE — `services.checklist.generator._extract_from_ast()`

**Evidence:** Spec pointer '/section_3/logging' has no covering checklist item

**Current source:**
```python
def _extract_from_ast(self, ast):
    ...

Question: How should _extract_from_ast be modified to eliminate this gap? Provide a concrete code modification, not a prose suggestion.


These prompts are written to `learning_artifacts/iter_N/improvement_prompts.md`
and can be pasted into any LLM chat, or auto-posted via `--ai-mode`.

---

## History Tracking

Every iteration appends to `learning_artifacts/history.json`:

```json
[
  {
    "iteration_id": "iter_001",
    "timestamp_utc": "2026-03-03T12:00:00+00:00",
    "corpus_score": 0.412,
    "mean_complete_pct": 0.031,
    "specs_met": 0,
    "specs_total": 52,
    "converged": false
  },
  ...
]

A rising corpus_score across iterations confirms that engine improvements are having the intended effect.

Monotonicity Assertion

tests/analysis/test_semantic_gradient.py::test_semantic_gradient_is_monotonic asserts that adding any single DSL section to the baseline spec never reduces checklist_preview_items. This is the formal guard against regressions in the generator's section-processing code paths.

Fuzz Suite

A second evaluation path runs the engine against a large population of random specs to check for regressions and edge-case failures.

# Run 500 random specs, report pass rate
python scripts/run_fuzz_suite.py --n 500 --seed 0

# Scale to 2000 specs and regenerate property tests
python scripts/run_fuzz_suite.py --n 2000 --seed 0 --gen-tests --tests-dir tests/property

The driver (scripts/run_fuzz_suite.py) calls spec_synthesizer.fuzz_corpus(n, seed) to produce deterministic random specs, then runs each through the engine and evaluates:

Assert Threshold
At least 1 checklist item emitted always
quality_ratio ≥ 0.80
completeness (tier 2+) ≥ 0.70

The generated tests/property/test_fuzz_property.py contains 500 parametrised pytest tests (one per seed) that re-assert these thresholds on every pytest run.

Current pass rate: 2000/2000 (100%).

Phase 2: Feature-Level Learning (Level 2)

Level 2 extends the same loop to feature-request inputs:

  1. Input: a feature fragment spec (e.g. "Add OAuth2 SSO to the product").
  2. Engine run: Engine.run_self_host(feature_spec) → targeted checklist.
  3. Evaluation: same sufficiency + quality pipeline but scoped to the feature's sections and pointers.
  4. Gap report: same gap taxonomy, but gaps are now feature-specific.
  5. Implementation test: a generated test that loads the full-product checklist and asserts the feature's acceptance criteria are present.

The key difference from Level 1: the "correct" convergence target for a feature request is not "all requirements covered" but "all acceptance criteria for the named feature are present in the product checklist". This requires a feature_acceptance_evaluator that matches feature-spec acceptance_criteria fields against product checklist items — a natural extension of requirements/completion.py.

Level 2 implementation is tracked separately.

Environment Variables Reference

Variable Effect
SHIELDCRAFT_SELFBUILD_ALLOW_DIRTY Set to 1 to bypass the repo sync gate
SHIELDCRAFT_LLM_ENDPOINT POST URL for --ai-mode prompt submission
SHIELDCRAFT_LLM_API_KEY Bearer token for SHIELDCRAFT_LLM_ENDPOINT
SPEC_TRIAL_BASELINE_DIR Baseline dir for diff tracking in run_spec_trials.py