Skip to content

Commit bcbd43a

Browse files
ProfRandom92ProfRandom92
authored
Add deterministic degraded coding workflow fixture
Add deterministic degraded coding workflow fixture - Add negative/degraded coding workflow fixture derived from the positive PR review workflow. - Validate expected ordering, reachability, causality, and invariant failures. - Update fixture template documentation for positive and negative fixture semantics. Validation reported in PR: targeted comparator, contract validator, positive fixture, negative fixture, full pytest, and npm run check passed.
1 parent 9cb78a4 commit bcbd43a

15 files changed

Lines changed: 422 additions & 0 deletions

‎docs/FIXTURE_TEMPLATE_v1.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -78,6 +78,13 @@ Recommended fields:
7878

7979
For positive fixtures, `expected_failures` and `allowed_failures` should be empty arrays.
8080

81+
## Positive and negative fixtures
82+
83+
- Positive fixtures should pass all must-hold contracts.
84+
- Negative/degraded fixtures should intentionally encode expected failures.
85+
- Expected failure labels must be committed in `expected/failures.json`.
86+
- Negative fixtures are for validating failure detection, not for runtime recovery.
87+
8188
## How fixtures are validated
8289

8390
A fixture validation test should:

‎fixtures/coding_workflow_pr_review_degraded_v1/README.md‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
# coding_workflow_pr_review_degraded_v1
2+
3+
Deterministic negative/degraded fixture for validating replay degradation detection in the ContractValidator + DependencyGraphComparator stack.
4+
5+
## Intentional degradations
6+
7+
1. **Ordering degradation**: reconstructed trace removes the `human_review` step and jumps from `run_tests`/`test_failure` to `merge`, breaking required ordering `generate_patch -> run_tests -> human_review -> merge`.
8+
2. **Reachability degradation**: reconstructed dependency graph removes recovery edges from `test_failure` to both `rollback` and `escalate_to_human`, violating `min_paths = 1` recovery expectations.
9+
3. **Causality degradation**: reconstructed dependency graph removes the causal edge `security_scan_failed -> deploy_blocked`.
10+
4. **Invariant degradation**: reconstructed dependency graph keeps `human_review` but removes all incoming dependencies, producing an orphan dependency condition.
11+
12+
## Expected ContractValidator failures
13+
14+
The fixture is expected to be inadmissible and to fail with:
15+
16+
- `POLICY_ORDER_BROKEN`
17+
- `RECOVERY_PATH_INVALID`
18+
- `CAUSAL_DEPENDENCY_LOSS`
19+
- `INVARIANT_VIOLATION`
20+
21+
## Comparator-level allowed failures
22+
23+
Relational checks may also emit comparator-level failure labels that are allowed for this degraded fixture:
24+
25+
- `ORPHAN_DEPENDENCY`
26+
- `DETACHED_DEPENDENCY`
27+
- `GRAPH_FRAGMENTATION`
28+
- `TEMPORAL_ORDER_VIOLATION`
29+
30+
This fixture is intentionally synthetic, deterministic, and scoped to replay-validation prototype behavior.

‎fixtures/coding_workflow_pr_review_degraded_v1/expected/admissibility.json‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
{
2+
"fixture_id": "coding_workflow_pr_review_degraded_v1",
3+
"fixture_version": "1.0.0",
4+
"expected_admissible": false,
5+
"must_fail_contracts": [
6+
"pre_merge_review",
7+
"recovery_path_available",
8+
"security_causal_block",
9+
"no_orphan_tool_calls"
10+
],
11+
"expected_failure_labels": [
12+
"POLICY_ORDER_BROKEN",
13+
"RECOVERY_PATH_INVALID",
14+
"CAUSAL_DEPENDENCY_LOSS",
15+
"INVARIANT_VIOLATION"
16+
],
17+
"expected_layer_scores": {
18+
"structural": 1.0,
19+
"relational": 0.0,
20+
"operational": 0.0,
21+
"governance": 1.0
22+
},
23+
"notes": "Negative degraded fixture for deterministic replay failure detection."
24+
}

‎fixtures/coding_workflow_pr_review_degraded_v1/expected/failures.json‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
{
2+
"expected_failures": [
3+
"POLICY_ORDER_BROKEN",
4+
"RECOVERY_PATH_INVALID",
5+
"CAUSAL_DEPENDENCY_LOSS",
6+
"INVARIANT_VIOLATION"
7+
],
8+
"allowed_failures": [
9+
"ORPHAN_DEPENDENCY",
10+
"DETACHED_DEPENDENCY",
11+
"GRAPH_FRAGMENTATION",
12+
"TEMPORAL_ORDER_VIOLATION"
13+
],
14+
"disallowed_failures": [
15+
"CYCLE_INTRODUCED",
16+
"REPLAY_NON_REPRODUCIBLE",
17+
"ARTIFACT_INTEGRITY_VIOLATION"
18+
]
19+
}

‎fixtures/coding_workflow_pr_review_degraded_v1/original/contracts/no_orphan_tool_calls.json‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
{
2+
"contract_id": "no_orphan_tool_calls",
3+
"layer": "relational",
4+
"type": "invariant",
5+
"definition": {
6+
"rule": "no_orphan_dependencies"
7+
},
8+
"severity": "HIGH"
9+
}

‎fixtures/coding_workflow_pr_review_degraded_v1/original/contracts/pre_merge_review.json‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
{
2+
"contract_id": "pre_merge_review",
3+
"layer": "operational",
4+
"type": "ordering",
5+
"definition": {
6+
"required_sequence": [
7+
"generate_patch",
8+
"run_tests",
9+
"human_review",
10+
"merge"
11+
]
12+
},
13+
"severity": "CRITICAL"
14+
}

‎fixtures/coding_workflow_pr_review_degraded_v1/original/contracts/recovery_path_available.json‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
{
2+
"contract_id": "recovery_path_available",
3+
"layer": "relational",
4+
"type": "reachability",
5+
"definition": {
6+
"from": "test_failure",
7+
"to": [
8+
"rollback",
9+
"escalate_to_human"
10+
],
11+
"min_paths": 1
12+
},
13+
"severity": "HIGH"
14+
}

‎fixtures/coding_workflow_pr_review_degraded_v1/original/contracts/security_causal_block.json‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
{
2+
"contract_id": "security_causal_block",
3+
"layer": "relational",
4+
"type": "causality",
5+
"definition": {
6+
"required_causal_edges": [
7+
["security_scan_failed", "deploy_blocked"]
8+
]
9+
},
10+
"severity": "HIGH"
11+
}

‎fixtures/coding_workflow_pr_review_degraded_v1/original/dependency_graph.json‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
{
2+
"graph_version": "1.0",
3+
"nodes": [
4+
{"node_id": "generate_patch", "label": "Generate patch", "metadata": {"phase": "build"}},
5+
{"node_id": "run_tests", "label": "Run tests", "metadata": {"phase": "verify"}},
6+
{"node_id": "test_failure", "label": "Test failure", "metadata": {"phase": "verify"}},
7+
{"node_id": "rollback", "label": "Rollback", "metadata": {"phase": "recovery"}},
8+
{"node_id": "security_scan_failed", "label": "Security scan failed", "metadata": {"phase": "security"}},
9+
{"node_id": "deploy_blocked", "label": "Deploy blocked", "metadata": {"phase": "security"}},
10+
{"node_id": "escalate_to_human", "label": "Escalate to human", "metadata": {"phase": "recovery"}},
11+
{"node_id": "human_review", "label": "Human review", "metadata": {"phase": "governance"}},
12+
{"node_id": "merge", "label": "Merge", "metadata": {"phase": "release"}}
13+
],
14+
"edges": [
15+
{"source": "generate_patch", "target": "run_tests", "relation": "PREREQUISITE", "metadata": {}},
16+
{"source": "run_tests", "target": "test_failure", "relation": "CAUSAL", "metadata": {}},
17+
{"source": "run_tests", "target": "security_scan_failed", "relation": "DATA_FLOW", "metadata": {}},
18+
{"source": "test_failure", "target": "rollback", "relation": "RECOVERY", "metadata": {}},
19+
{"source": "test_failure", "target": "escalate_to_human", "relation": "RECOVERY", "metadata": {}},
20+
{"source": "security_scan_failed", "target": "deploy_blocked", "relation": "CAUSAL", "metadata": {}},
21+
{"source": "rollback", "target": "human_review", "relation": "TEMPORAL", "metadata": {}},
22+
{"source": "escalate_to_human", "target": "human_review", "relation": "TEMPORAL", "metadata": {}},
23+
{"source": "human_review", "target": "merge", "relation": "PREREQUISITE", "metadata": {}},
24+
{"source": "run_tests", "target": "merge", "relation": "PREREQUISITE", "metadata": {}},
25+
{"source": "deploy_blocked", "target": "merge", "relation": "BLOCKER", "metadata": {"state": "prevented"}}
26+
]
27+
}

‎fixtures/coding_workflow_pr_review_degraded_v1/original/state.json‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
{
2+
"evidence": {
3+
"pr_id": "PR-122-fixture",
4+
"test_suite": "unit",
5+
"security_gate": "required"
6+
},
7+
"constraints": {
8+
"requires_human_review": true,
9+
"requires_clean_tests_before_merge": true
10+
},
11+
"blockers": [
12+
"test_failure",
13+
"security_scan_failed",
14+
"deploy_blocked"
15+
],
16+
"recovery_paths": {
17+
"test_failure": [
18+
"rollback",
19+
"escalate_to_human"
20+
]
21+
},
22+
"dependencies": {
23+
"merge": [
24+
"human_review",
25+
"run_tests"
26+
]
27+
}
28+
}

0 commit comments

Comments
 (0)