Skip to content

Commit caf9431

Browse files
committed
docs: codify enforcement-strength test authoring standard
1 parent 5153d41 commit caf9431

4 files changed

Lines changed: 86 additions & 0 deletions

File tree

CONTRIBUTING.md

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ Thanks for contributing.
88
- Keep ClawZero positioned as an in-path enforcement substrate.
99
- Do not reframe the project as attack-simulation-first.
1010
- Maintain witness schema compatibility.
11+
- Follow the enforcement-strength test standard in `docs/test-authoring-guide.md`.
1112

1213
## Development
1314

@@ -16,6 +17,8 @@ pytest tests/test_claims.py -v
1617
python demo/openclaw_attack_demo.py --mode compare --scenario shell
1718
```
1819

20+
For test changes, run targeted suites locally and include output in the PR.
21+
1922
## Pull Requests
2023

2124
Please include:
@@ -24,3 +27,12 @@ Please include:
2427
- design approach
2528
- tests and verification output
2629
- contract/schema impact (if any)
30+
31+
## Test Quality Gate
32+
33+
PRs that add or modify tests must satisfy:
34+
35+
- no weak assertions (existence-only / no-op assertions)
36+
- explicit enforcement-path assertions (`decision`, `reason_code`, `sink_type`)
37+
- witness/session assertions where the feature depends on them
38+
- behavior grounded in actual runtime contracts (not aspirational assumptions)

docs/index.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,4 +30,5 @@ Use the attack demo as **proof of enforcement**, not as the product center.
3030
## Operator Docs
3131

3232
- [Integration Quickstarts](integration-quickstarts.md)
33+
- [Test Authoring Guide](test-authoring-guide.md)
3334
- [Claims Registry](claims-registry.md)

docs/test-authoring-guide.md

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
# Test Authoring Guide
2+
3+
This is the enforcement-strength standard for all new ClawZero tests.
4+
5+
## Non-Negotiable Standard
6+
7+
Every test must enforce behavior, not just execution.
8+
9+
- No weak assertions like only `is not None`, existence-only checks, or broad `in` checks when exact behavior is known.
10+
- No tolerance paths that silently accept both `allow` and `block` unless the contract explicitly allows both and each branch has strict assertions.
11+
- No assumptions about engine behavior. Assertions must match documented runtime contracts and current policy semantics.
12+
13+
## Required Assertions by Path
14+
15+
### Block Path
16+
17+
Use `pytest.raises(ExecutionBlocked)` and assert:
18+
19+
- `decision.decision == "block"`
20+
- `decision.sink_type == expected_sink`
21+
- `decision.reason_code == expected_reason` (or documented bounded set only where contract requires)
22+
23+
### Allow / Annotate Path
24+
25+
Assert all of:
26+
27+
- Returned result semantics (exact expected payload/shape when deterministic)
28+
- Witness sink, decision, and reason code
29+
- Provenance contract fields when applicable (`taint_level`, markers, source chain)
30+
31+
## Session / Chain Tests
32+
33+
For multi-step/session tests, assert:
34+
35+
- Chain detections include expected pattern(s)
36+
- Detection evidence references real request IDs from the executed chain
37+
- Threshold-sensitive behavior is validated against profile thresholds
38+
- Session report counts and persisted log contents match executed calls
39+
40+
## Witness Assertions
41+
42+
When a test depends on witness artifacts, assert:
43+
44+
- witness exists and is a dict
45+
- witness request linkage (`request_id`)
46+
- decision/sink/reason match expected enforcement outcome
47+
- provenance fields are validated against engine normalization rules
48+
49+
## Generated Test Files
50+
51+
Generated suites are held to the same bar as handwritten suites.
52+
53+
- No exception-assertion no-ops
54+
- No count-inflation-only assertions
55+
- Same strict enforcement/result/witness/session checks as non-generated tests
56+
57+
## Review Checklist (PR Gate)
58+
59+
Before merge, reviewers should verify:
60+
61+
- Weak assertion patterns are absent
62+
- Enforcement path(s) are explicitly required by assertions
63+
- Reason codes and sinks are validated, not implied
64+
- Tests are grounded in current contracts, not aspirational behavior
65+
- Local targeted run and CI are both green
66+
67+
## Useful Contract Anchors
68+
69+
- `tests/policy_matrix_data.py`
70+
- `tests/test_policy_matrix_generated.py`
71+
- `tests/runtime/test_engine_parity_contract.py`
72+
- `tests/test_witness_integrity_matrix.py`

mkdocs.yml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@ nav:
1717
- Policies: policies.md
1818
- OpenClaw Integration: openclaw-integration.md
1919
- Integration Quickstarts: integration-quickstarts.md
20+
- Test Authoring Guide: test-authoring-guide.md
2021
- Claims Registry: claims-registry.md
2122
- Release Checklist: release-checklist.md
2223
- GitHub Launch Copy: github-profile-copy.md

0 commit comments

Comments
 (0)