Skip to content

CHANGELOG.md

Latest commit

 

History

History
35 lines (29 loc) · 3.58 KB

CHANGELOG.md

File metadata and controls

35 lines (29 loc) · 3.58 KB
Raw

Changelog

Unreleased

Fuzz evaluation suite (2026-03-03)

  • Add scripts/run_fuzz_suite.py: generates N random specs, runs engine evaluation loop, reports quality/completeness pass rate and failure patterns.
  • Add scripts/spec_synthesizer.py --fuzz N: generates random specs seeded from a 90-requirement vocabulary across 9 domains (auth, data, API, governance, reliability, privacy, security, performance, observability).
  • Add tests/property/test_fuzz_property.py: 500 auto-generated parametrised pytest tests, one per seed, asserting quality ≥ 80% and completeness ≥ 70%.
  • Fuzz results: 2000/2000 random specs pass (100%) at 20s runtime.

Checklist quality and completeness improvements (2026-03-03)

  • P0: Skip the prose interpreter for JSON specs; use _dsl_items() semantic templates instead, eliminating structural token junk ({, }, ],) from generated checklists.
  • P1: Add _dsl_items() generator that produces well-formed imperative items from all DSL fields: invariants, sections, agents, artifact_contract, execution.steps, instructions, codegen_targets, determinism.
  • P2: Promote checklist items to HIGH confidence when they contain strong normative keywords (must, must not, never, refuse, unsafe). Preserve pre-set confidence from DSL generators.
  • P3: Expand COMMON_IMPERATIVE_VERBS in checklist/quality.py from 25 to 38 verbs (added assert, complete, confirm, detect, disable, enable, execute, generate, prevent, reject, set, test, write).
  • Quality fix: Accept checklist items as actionable when they contain normative keywords (must, shall, never, requires) even if the first word is not in the imperative-verb whitelist.
  • Completeness fix: _build_minimal_completeness now checks text + value + action + claim combined for token overlap rather than action alone.
  • Token normaliser fix: Replace all non-alphanumeric characters with spaces in tokeniser so manifest.json splits to {manifest, json} instead of {manifestjson}.
  • Stale artifact fix: _run_single_spec() clears stale build artifacts before each run to prevent old checklist.json from shadowing newly generated checklist_draft.json.
  • Requirements extraction fix: JSON specs now extract requirements semantically from DSL fields (sections.description, invariants, agents, artifact_contract, execution.steps, instructions) instead of running the prose extractor on raw JSON text.

Learning loop convergence (2026-03-03)

  • Learning trial corpus_score rose from 0.6210 to 1.0000 (32/32 specs met, converged=True) after the above fixes.
  • Generated tests in tests/generated/: all 229 tests pass.

Self-learning test strategy — Level 1 (2026-02-xx)

  • Add scripts/run_learning_trial.py: orchestrator for iterative spec-corpus evaluation with gap analysis, improvement prompts, and auto-generated pytest tests.
  • Add scripts/spec_synthesizer.py: generates 32 fixed spec variants across 4 tiers (prose, structured, partial-DSL, full-DSL).
  • Add src/shieldcraft/learning/: gap analyzer, code suggestion formatter, dynamic test generator.
  • Add tests/generated/: auto-generated regression + improvement-backlog tests.
  • Add docs/LEARNING_STRATEGY.md.

Governance and determinism mapping (2025-xx-xx)

  • Add deterministic governance mapping (src/shieldcraft/services/governance/map.py) linking docs to enforcement rules and enabling auditability.
  • Add src/shieldcraft/interpreter/__init__.py _is_json_structural() safety filter to prevent JSON structural tokens from becoming checklist items.