test(feature-model): fuzz the constraint solver (REQ-266)#715
Merged
Conversation
The proptest strategies hard-coded `constraints: vec![]`, so the solver's highest-risk code — cross-tree constraint evaluation, `implies` propagation, `excludes`, and REQ-257's fail-loud gate — had ZERO property-test coverage. Add: - A constraint-generating strategy (implies/excludes/and/or/not over real feature-name leaves) kept as a typed value so the property can evaluate each constraint independently — an oracle to grade the solver against. - prop_ok_means_constraints_actually_hold: if solve() returns Ok, every constraint genuinely holds under the resolved selection. This is the anti-silent-pass property — a return to the pre-REQ-257 `_ => true` blind spot would surface here as an Ok whose selection violates a constraint. - prop_implies_propagation + prop_solver_never_panics_with_constraints. - Deterministic fail-loud tests: an unevaluatable attribute comparison yields UnevaluatableConstraint (REQ-257); a constraint naming a nonexistent feature yields UnknownConstraintFeature (REQ-259). Also surfaces a real latent bug: a shared child reachable through two parents (a diamond / DAG) is mis-reported as a cycle by validate_tree's BFS (global-visited-as-cycle). Filed as REQ-269 with a regression test (diamond_shared_child_is_not_a_cycle, #[ignore]d until the fix); confirmed failing with "cycle detected involving feature `shared`". Confirmed with `cargo test -p rivet-core --test proptest_feature_model` (12 passed, 1 ignored) and clippy --all-targets clean. Test-only + artifact change; the wasm seam is untouched. Implements: REQ-266 Verifies: REQ-257, REQ-259 Refs: REQ-269, DD-076
📐 Rivet artifact delta
Graphgraph LR
REQ_266["REQ-266"]:::modified
REQ_269["REQ-269"]:::added
classDef added fill:#d4edda,stroke:#28a745,color:#155724
classDef removed fill:#f8d7da,stroke:#dc3545,color:#721c24
classDef modified fill:#fff3cd,stroke:#ffc107,color:#856404
classDef overflow fill:#e2e3e5,stroke:#6c757d,color:#495057,stroke-dasharray: 3 3
Added
Modified
Posted by |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
There was a problem hiding this comment.
⚠️ Performance Alert ⚠️
Possible performance regression was detected for benchmark 'Rivet Criterion Benchmarks'.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 1.20.
| Benchmark suite | Current: 6480063 | Previous: 8abbb62 | Ratio |
|---|---|---|---|
store_insert/100 |
86586 ns/iter (± 913) |
69126 ns/iter (± 437) |
1.25 |
store_insert/1000 |
928388 ns/iter (± 9279) |
740783 ns/iter (± 5800) |
1.25 |
store_lookup/100 |
1947 ns/iter (± 8) |
1522 ns/iter (± 25) |
1.28 |
store_lookup/1000 |
25050 ns/iter (± 173) |
18584 ns/iter (± 46) |
1.35 |
store_lookup/10000 |
361448 ns/iter (± 9453) |
265291 ns/iter (± 954) |
1.36 |
store_by_type/100 |
96 ns/iter (± 0) |
74 ns/iter (± 0) |
1.30 |
store_by_type/1000 |
95 ns/iter (± 0) |
74 ns/iter (± 0) |
1.28 |
store_by_type/10000 |
95 ns/iter (± 0) |
74 ns/iter (± 0) |
1.28 |
schema_load_and_merge |
1525307 ns/iter (± 29137) |
1181005 ns/iter (± 22486) |
1.29 |
link_graph_build/100 |
166807 ns/iter (± 1805) |
126830 ns/iter (± 655) |
1.32 |
link_graph_build/1000 |
1985681 ns/iter (± 5978) |
1495979 ns/iter (± 9916) |
1.33 |
validate/100 |
456973 ns/iter (± 6255) |
341875 ns/iter (± 3801) |
1.34 |
validate/1000 |
15016997 ns/iter (± 104351) |
11276807 ns/iter (± 269977) |
1.33 |
validate/10000 |
1101243769 ns/iter (± 20667852) |
762756012 ns/iter (± 8858190) |
1.44 |
traceability_matrix/100 |
4274 ns/iter (± 331) |
3215 ns/iter (± 15) |
1.33 |
traceability_matrix/1000 |
45133 ns/iter (± 271) |
34302 ns/iter (± 168) |
1.32 |
traceability_matrix/10000 |
734981 ns/iter (± 9516) |
555669 ns/iter (± 4285) |
1.32 |
diff/100 |
64047 ns/iter (± 197) |
46183 ns/iter (± 131) |
1.39 |
diff/1000 |
722526 ns/iter (± 13596) |
521705 ns/iter (± 2750) |
1.38 |
diff/10000 |
8238501 ns/iter (± 141364) |
6163298 ns/iter (± 202417) |
1.34 |
query/100 |
1120 ns/iter (± 2) |
904 ns/iter (± 2) |
1.24 |
query/1000 |
15042 ns/iter (± 35) |
11284 ns/iter (± 99) |
1.33 |
query/10000 |
225366 ns/iter (± 3195) |
172570 ns/iter (± 970) |
1.31 |
document_parse/10 |
21752 ns/iter (± 97) |
17012 ns/iter (± 124) |
1.28 |
document_parse/100 |
153972 ns/iter (± 1044) |
117738 ns/iter (± 295) |
1.31 |
document_parse/1000 |
1393310 ns/iter (± 22561) |
1102249 ns/iter (± 26378) |
1.26 |
This comment was automatically generated by workflow using github-action-benchmark.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
REQ-266 — fuzz the constraint solver (v0.29.0)
The proptest strategies hard-coded
constraints: vec![], so the solver'shighest-risk code — cross-tree constraint evaluation,
impliespropagation,
excludes, and REQ-257's fail-loud gate — had zeroproperty-test coverage. This adds it.
What's covered
implies/excludes/and/or/notover real feature-name leaves, kept as a typed
TestConstraintso theproperty can evaluate each constraint independently of the solver (an
oracle, so the solver doesn't grade its own homework).
prop_ok_means_constraints_actually_hold— the anti-silent-passproperty: if
solve()returnsOk, every constraint genuinely holdsunder the resolved selection. A regression to the pre-REQ-257
_ => trueblind spot would surface here as an
Okwhose selection violates aconstraint.
prop_implies_propagation,prop_solver_never_panics_with_constraints.UnevaluatableConstraint(REQ-257); constraint naming a nonexistentfeature →
UnknownConstraintFeature(REQ-259).Bug surfaced → REQ-269
The shared-child coverage revealed a real latent bug: a feature reachable
through two parents (a diamond / DAG) is mis-reported as a cycle by
validate_tree's BFS (global-visited-as-cycle). Confirmed failing withcycle detected involving featureshared``. Filed as REQ-269 (v0.29.0)with a regression test
diamond_shared_child_is_not_a_cycle(`#[ignore]`duntil the fix lands, then flip to enforced).
Verification
cargo test -p rivet-core --test proptest_feature_model→ 12 passed, 1ignored.
clippy --all-targets(1.97.0) clean. Test-only + artifactchange — the composition core and wasm seam are untouched.
Implements: REQ-266 · Verifies: REQ-257, REQ-259 · Refs: REQ-269, DD-076
🤖 Generated with Claude Code