|
| 1 | +# Enterprise-Readiness Test Harness |
| 2 | + |
| 3 | +## Status |
| 4 | + |
| 5 | +**Version:** v0.1 |
| 6 | +**Scope:** Synthetic / mocked downstream systems / CI replay |
| 7 | +**Claim:** Test-harness evidence only |
| 8 | + |
| 9 | +This document defines what the enterprise-shaped scenario harness currently proves and what remains absent. |
| 10 | + |
| 11 | +## Clean claim |
| 12 | + |
| 13 | +This harness tests whether a bounded AI-assisted action can reach a mocked downstream system when required authority is missing. |
| 14 | + |
| 15 | +For ESP-001, the answer is: |
| 16 | + |
| 17 | +```text |
| 18 | +missing authority_token |
| 19 | + -> DENY |
| 20 | + -> mock email adapter not called |
| 21 | + -> downstream_send=false |
| 22 | + -> receipt_written=true |
| 23 | +``` |
| 24 | + |
| 25 | +## What this harness proves |
| 26 | + |
| 27 | +On the synthetic path, the harness proves: |
| 28 | + |
| 29 | +- the action class is named: `SEND_EXTERNAL_EMAIL` |
| 30 | +- the required authority field is missing: `authority_token` |
| 31 | +- the gate returns `DENY` |
| 32 | +- the mocked downstream email adapter is not called |
| 33 | +- the adapter records zero sent messages |
| 34 | +- a receipt is generated |
| 35 | +- the scenario can be replayed by pytest |
| 36 | +- the test can run in GitHub Actions |
| 37 | + |
| 38 | +## What this harness does not prove |
| 39 | + |
| 40 | +This harness does not prove: |
| 41 | + |
| 42 | +- live SMTP / API non-execution |
| 43 | +- production runtime enforcement |
| 44 | +- enterprise deployment |
| 45 | +- certification |
| 46 | +- compliance |
| 47 | +- legal adequacy |
| 48 | +- closure of all bypass paths |
| 49 | +- no dispatch through queues, retries, alternate APIs, human handoff, or connector side channels |
| 50 | +- real-world controlled application |
| 51 | +- external review |
| 52 | + |
| 53 | +## Required evidence for stronger claims |
| 54 | + |
| 55 | +| Stronger claim | Required evidence | |
| 56 | +|---|---| |
| 57 | +| Live runtime enforcement | Gate integrated with a real execution layer or realistic service boundary | |
| 58 | +| Downstream non-execution | External connector call logs or service-level event evidence | |
| 59 | +| Path-universal control | Architecture showing every consequence path routes through the gate or is explicitly out of scope | |
| 60 | +| Enterprise readiness | Multiple realistic enterprise workflows, CI replay, bypass tests, external review, and controlled organisational pilot evidence | |
| 61 | +| Production readiness | Deployment architecture, operational monitoring, failure handling, persistence, security review, and live audit trail | |
| 62 | +| Certification / compliance | Independent qualified assessment against a named framework or legal requirement | |
| 63 | + |
| 64 | +## Current label |
| 65 | + |
| 66 | +Safe: |
| 67 | + |
| 68 | +```text |
| 69 | +enterprise-readiness test harness |
| 70 | +``` |
| 71 | + |
| 72 | +Unsafe: |
| 73 | + |
| 74 | +```text |
| 75 | +enterprise-ready system |
| 76 | +enterprise deployment evidence |
| 77 | +production enforcement proof |
| 78 | +certified control |
| 79 | +compliance-ready system |
| 80 | +``` |
| 81 | + |
| 82 | +## Next hardening path |
| 83 | + |
| 84 | +1. Add more mocked consequence classes: |
| 85 | + - payment instruction |
| 86 | + - access change |
| 87 | + - HR decision support |
| 88 | + - vendor risk escalation |
| 89 | + |
| 90 | +2. Add bypass tests: |
| 91 | + - retry cannot dispatch after denial |
| 92 | + - queue cannot dispatch after denial |
| 93 | + - stale DecisionRecord rejected |
| 94 | + - alternate send path blocked |
| 95 | + - audit failure fails closed |
| 96 | + |
| 97 | +3. Add evidence matrix: |
| 98 | + - claim |
| 99 | + - required evidence |
| 100 | + - current proof |
| 101 | + - missing proof |
| 102 | + - next test |
| 103 | + |
| 104 | +4. Add external review route: |
| 105 | + - issue template for hostile review |
| 106 | + - reviewer checklist |
| 107 | + - reproducibility instructions |
| 108 | + |
| 109 | +## Stop rule |
| 110 | + |
| 111 | +Do not call this enterprise-ready until an external reviewer or bounded organisation can inspect, run, and challenge the harness against a realistic workflow. |
0 commit comments