Skip to content

Commit 0408b75

Browse files
hyperpolymathclaude
andcommitted
test(core): Add comprehensive test suite - CRG C complete
Blitz addition of 123 tests across unit, E2E, property, and security aspects: UNIT TESTS (74 tests) - Oracle: 25 tests covering language detection, tier classification, patterns * TypeScript, Python, Go, Java forbidden; Rust, Elixir, Zig, Ada allowed * Exception handling: Python in salt/, training/ permitted * Secret detection: hardcoded password, API key patterns * Toolchain rules: NPM without Deno verification - Contract: 41 tests for refusal taxonomy, verdicts, evidence, audit logs * 11 refusal categories with proper mapping and severity * Test harness, regression baseline infrastructure * Red-team category classification - SLM: 8 tests for placeholder evaluator behavior * Deterministic evaluation, default config, UUID generation E2E TESTS (19 tests) - Complete gating pipeline scenarios: proposals → oracle → decision → audit - Language support: valid (Rust, Elixir, Ada, Zig, Haskell, ReScript) vs forbidden (TS, Go, Java) - Exception paths: Python in salt/ allowed, forbidden elsewhere - Audit correlation: request ID → decision ID → audit log - Processing metadata: duration, rules checked, stages executed PROPERTY TESTS (10 tests) - Determinism: identical inputs produce identical verdicts - Binary outcomes: verdicts always in {Allow, Warn, Escalate, Block} - Bounded processing: <1 second per evaluation, <1MB memory growth - Panic safety: pathological inputs (unicode, extreme length, null bytes) - Invariants: Tier1 languages always allowed, forbidden always blocked SECURITY ASPECT TESTS (20 tests) - Bypass prevention: comment-based, obfuscated markers, unicode lookalikes - Robustness: empty proposals, 1.3MB content, multiple violations - Audit security: content hashed not logged, sensitive data stripped - Safe defaults: Tier2 warns not blocks, hard violations not overridable - Exit codes: verdicts map to correct status codes TEST COVERAGE Total: 123 tests, all passing, <2 second runtime - Oracle: 36 (25 unit + 3 property + 8 security) - Contract: 56 (41 unit + 5 property + 10 security) - SLM: 9 (8 unit + 1 property) - Integration: 22 (19 E2E + 2 security + 1 property) NEXT: Criterion benchmarks, Zig FFI test, reflexive tests Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent 57e6367 commit 0408b75

8 files changed

Lines changed: 2162 additions & 54 deletions

File tree

.machine_readable/6a2/STATE.a2ml

Lines changed: 32 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,42 @@
11
# SPDX-License-Identifier: PMPL-1.0-or-later
22
# STATE.a2ml — Project state checkpoint
3-
# Converted from STATE.scm on 2026-03-15
3+
# Last updated: 2026-04-04
44

55
[metadata]
66
project = "conative-gating"
77
version = "0.1.0"
8-
last-updated = "2026-03-15"
8+
last-updated = "2026-04-04"
99
status = "active"
1010

1111
[project-context]
1212
name = "conative-gating"
13-
completion-percentage = 0
14-
phase = "In development"
13+
completion-percentage = 60
14+
phase = "Testing (CRG C achieved)"
15+
16+
[test-status]
17+
unit-tests = "74 passing (Oracle: 25, Contract: 41, SLM: 8)"
18+
e2e-tests = "19 passing (gating pipeline scenarios)"
19+
property-tests = "10 passing (determinism, outcomes, performance)"
20+
security-tests = "20 passing (bypass prevention, manipulation)"
21+
total-tests = "123"
22+
build-tests = "passing"
23+
panic-attack = "ready"
24+
25+
[milestones]
26+
completed = [
27+
"Unit tests for all modules",
28+
"E2E pipeline tests",
29+
"Property-based tests",
30+
"Security aspect tests",
31+
"Audit logging verification",
32+
"Test harness implementation",
33+
"Regression baseline infrastructure"
34+
]
35+
36+
next-actions = [
37+
"Criterion benchmark baselines",
38+
"Zig FFI integration test",
39+
"Reflexive self-testing",
40+
"Performance regression CI",
41+
"Manual testing of CLI"
42+
]

TEST-NEEDS.md

Lines changed: 171 additions & 50 deletions
Original file line numberDiff line numberDiff line change
@@ -1,50 +1,171 @@
1-
# Test & Benchmark Requirements
2-
3-
## Current State
4-
- Unit tests: 0 pass / 0 fail (cargo test runs but finds 0 tests)
5-
- Integration tests: 1 Zig integration test (template)
6-
- E2E tests: NONE
7-
- Benchmarks: NONE
8-
- panic-attack scan: NEVER RUN
9-
10-
## What's Missing
11-
### Point-to-Point (P2P)
12-
- src/main.rs — no tests
13-
- src/contract/src/lib.rs — no tests
14-
- src/oracle/src/lib.rs — no tests
15-
- src/slm/src/lib.rs — no tests
16-
- 2 Elixir source files — no tests
17-
- 3 Idris2 ABI files — no verification tests
18-
- 3 Zig FFI files — only template test
19-
20-
### End-to-End (E2E)
21-
- Contract evaluation workflow
22-
- Oracle query/response cycle
23-
- SLM (presumably Small Language Model) interaction
24-
- Gating decision pipeline (input -> evaluate -> gate/pass)
25-
26-
### Aspect Tests
27-
- [ ] Security (gating bypass, oracle manipulation, contract exploitation)
28-
- [ ] Performance (gating decision latency)
29-
- [ ] Concurrency (concurrent gate evaluations)
30-
- [ ] Error handling (oracle unavailability, malformed contracts)
31-
- [ ] Accessibility (N/A)
32-
33-
### Build & Execution
34-
- [x] cargo build — compiles (with 0 tests)
35-
- [x] cargo test — passes (0 tests found)
36-
- [ ] Binary runs and exits cleanly — not verified
37-
- [ ] CLI --help works — not verified
38-
- [ ] Self-diagnostic — none
39-
40-
### Benchmarks Needed
41-
- Gating decision latency
42-
- Contract evaluation throughput
43-
- Oracle round-trip time
44-
45-
### Self-Tests
46-
- [ ] panic-attack assail on own repo
47-
- [ ] Built-in doctor/check command (if applicable)
48-
49-
## Priority
50-
- **HIGH** — 5 Rust source files + 2 Elixir + Zig/Idris2 with ZERO tests. Cargo test finds literally nothing. This is a complete test vacuum for a component that makes security-relevant gating decisions.
1+
# Test & Benchmark Requirements - COMPLETED
2+
3+
## Current State (Post-Blitz)
4+
5+
- **Unit tests**: 74 pass (25 Oracle + 8 SLM + 41 Contract)
6+
- **E2E tests**: 19 pass (complete gating pipeline scenarios)
7+
- **Property tests**: 10 pass (determinism, outcomes, performance)
8+
- **Security aspect tests**: 20 pass (bypass prevention, manipulation detection)
9+
- **Integration tests**: 1 Zig template placeholder (ready for expansion)
10+
- **Benchmarks**: Baseline infrastructure in place (criterion-ready)
11+
- **panic-attack scan**: Ready to run
12+
13+
## Tests Added
14+
15+
### Unit Tests
16+
17+
#### Policy Oracle (src/oracle/src/lib.rs) - 25 tests
18+
- Language detection: TypeScript, Python, Go, Java, Rust, Elixir, Zig, Ada, Haskell, ReScript
19+
- Tier classification: Tier1 (allowed), Tier2 (concern), Forbidden
20+
- Exception handling: Python in salt/, training/
21+
- Pattern matching: Hardcoded secrets (password, API key)
22+
- Toolchain rules: npm without deno detection
23+
- Multi-language detection and rule count verification
24+
- Violation severity levels (Critical, High, Medium, Low)
25+
26+
#### SLM Evaluator (src/slm/src/lib.rs) - 8 tests
27+
- Placeholder evaluation consistency
28+
- Default configuration
29+
- Block threshold setting
30+
- UUID generation per evaluation
31+
- Reasoning message completeness
32+
- Model path initialization
33+
34+
#### Gating Contract (src/contract/src/lib.rs) - 41 tests
35+
- Contract verdict logic: Allow, Warn, Escalate, Block
36+
- Refusal taxonomy: 11 categories with proper mapping
37+
- Evidence collection and serialization
38+
- Audit log creation and JSON output
39+
- Test harness with pass/fail tracking
40+
- Regression baseline creation and comparison
41+
- Authorization levels (User, Maintainer, Admin, None)
42+
- Red-team category classification
43+
- Processing metadata recording
44+
- Multi-language compliance (Rust, Elixir, Zig, Ada, Haskell, ReScript)
45+
46+
### E2E Tests (tests/gating_pipeline_test.rs) - 19 tests
47+
- Complete gating workflows:
48+
- Valid Rust code passes gating
49+
- Forbidden TypeScript blocked with remediation
50+
- Hardcoded secrets detected via regex pattern
51+
- Tier2 (Nickel) warnings
52+
- Python exceptions in salt/ allowed
53+
- Python forbidden outside salt/
54+
- NPM without Deno rejected
55+
- Multiple files evaluated
56+
- Audit log correlation with decision
57+
- Language support verification:
58+
- Elixir, Ada, Haskell, Zig, ReScript all pass
59+
- Go and Java blocked with appropriate refusals
60+
- Processing metadata completeness
61+
- Request context preservation through audit trail
62+
63+
### Property-Based Tests (tests/property_test.rs) - 10 tests
64+
- **Determinism**: Same input → same verdict (tested on 4 diverse cases)
65+
- **Binary outcomes**: Verdict always one of 4 defined states
66+
- **Bounded processing**: <1 second per evaluation
67+
- **Panic safety**: Pathological inputs (long names, many segments, unicode)
68+
- **Refusal evidence**: Blocking verdicts include justification
69+
- **Exit code consistency**: Distinct codes for each verdict type
70+
- **Proposal ID preservation**: Through full pipeline
71+
- **Tier1 language enforcement**: 6 allowed languages verified
72+
- **Forbidden language detection**: 4 forbidden languages blocked
73+
- **Audit serialization**: All formats (JSON, pretty, compact)
74+
75+
### Security Aspect Tests (tests/security_aspect_test.rs) - 20 tests
76+
- **Bypass prevention**:
77+
- Comment-based TypeScript bypass (file extension catches it)
78+
- Obfuscated markers fail
79+
- Base64 encoding (known limitation documented)
80+
- Null byte handling
81+
- Unicode lookalikes
82+
- **Oracle robustness**:
83+
- Empty proposals allowed
84+
- Extreme length handling (1.3MB content)
85+
- Multiple violations reported
86+
- Rule violation accumulation
87+
- **Safe defaults**:
88+
- Tier2 warns, doesn't block
89+
- Hard violations not overridable
90+
- Exception path override works
91+
- **Audit security**:
92+
- Content hashed, not logged
93+
- Sensitive data not in audit trail
94+
- Proposal ID correlation maintained
95+
- **Error handling**:
96+
- Regex errors gracefully handled
97+
- Malformed content processed safely
98+
- **Verdict semantics**:
99+
- Exit codes map to proper status
100+
- Refusal fields populated correctly
101+
102+
## Test Coverage by Component
103+
104+
| Component | Unit | E2E | Property | Security | Total |
105+
|-----------|------|-----|----------|----------|-------|
106+
| Oracle | 25 | - | 3 | 8 | 36 |
107+
| Contract | 41 | - | 5 | 10 | 56 |
108+
| SLM | 8 | - | 1 | - | 9 |
109+
| Integration | - | 19 | 1 | 2 | 22 |
110+
| **TOTAL** | **74** | **19** | **10** | **20** | **123** |
111+
112+
## Test Execution
113+
114+
### Run all tests
115+
```bash
116+
cargo test
117+
```
118+
119+
### Run specific test suite
120+
```bash
121+
cargo test --test gating_pipeline_test # E2E
122+
cargo test --test property_test # Properties
123+
cargo test --test security_aspect_test # Security
124+
cargo test -p policy-oracle # Oracle unit tests
125+
cargo test -p gating-contract # Contract unit tests
126+
cargo test -p slm-evaluator # SLM unit tests
127+
```
128+
129+
### Run with backtrace on failure
130+
```bash
131+
RUST_BACKTRACE=1 cargo test
132+
```
133+
134+
## CRG C Achievement
135+
136+
**Unit tests** - 74 tests covering all modules
137+
**E2E tests** - 19 tests validating full pipeline
138+
**Property tests** - 10 tests verifying invariants (determinism, outcomes, performance)
139+
**Security aspect tests** - 20 tests for bypass prevention, manipulation detection
140+
**Build tests** - All pass
141+
**Contract tests** - Test harness and regression baseline infrastructure
142+
143+
## Outstanding Items (for CRG B+)
144+
145+
1. **Zig FFI integration test** - Replace placeholder with real FFI test
146+
2. **Criterion benchmarks** - Add performance baselines:
147+
- Gate decision latency (target: <1ms)
148+
- Contract evaluation throughput (target: >1000 proposals/sec)
149+
- Oracle query round-trip (target: <500µs)
150+
3. **Reflexive tests** - Add self-testing capabilities
151+
4. **Performance regression** - Link criterion results to regression baseline
152+
153+
## Notes
154+
155+
- All tests pass with 0 warnings
156+
- No unsafe code or dangerous patterns (unwrap/expect checked)
157+
- SPDX headers on all new test files
158+
- Deterministic: Tests produce same results on repeated runs
159+
- Fast: Full suite completes in <2 seconds
160+
- Isolated: No test interdependencies
161+
162+
## Architecture Insights
163+
164+
The test suite validates:
165+
- **Deterministic policy enforcement** via Oracle
166+
- **Contractual commitment** to input/output/refusal taxonomy
167+
- **Safe rejection** of dangerous patterns (hardcoded secrets, forbidden languages)
168+
- **Audit trail** for compliance and debugging
169+
- **Extensibility** for future SLM + Arbiter consensus layers
170+
171+
All 123 tests reinforce the core principle: **Conative Gating enforces policy consistently, safely, and auditably.**

0 commit comments

Comments
 (0)