Skip to content

test(feature-model): fuzz the constraint solver (REQ-266)#715

Merged
avrabe merged 1 commit into
mainfrom
feat/req-266-constraint-proptests
Jul 16, 2026
Merged

test(feature-model): fuzz the constraint solver (REQ-266)#715
avrabe merged 1 commit into
mainfrom
feat/req-266-constraint-proptests

Conversation

@avrabe

@avrabe avrabe commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

REQ-266 — fuzz the constraint solver (v0.29.0)

The proptest strategies hard-coded constraints: vec![], so the solver's
highest-risk code — cross-tree constraint evaluation, implies
propagation, excludes, and REQ-257's fail-loud gate — had zero
property-test coverage. This adds it.

What's covered

  • Constraint-generating strategyimplies/excludes/and/or/not
    over real feature-name leaves, kept as a typed TestConstraint so the
    property can evaluate each constraint independently of the solver (an
    oracle, so the solver doesn't grade its own homework).
  • prop_ok_means_constraints_actually_hold — the anti-silent-pass
    property: if solve() returns Ok, every constraint genuinely holds
    under the resolved selection. A regression to the pre-REQ-257 _ => true
    blind spot would surface here as an Ok whose selection violates a
    constraint.
  • prop_implies_propagation, prop_solver_never_panics_with_constraints.
  • Deterministic fail-loud tests: unevaluatable attribute comparison →
    UnevaluatableConstraint (REQ-257); constraint naming a nonexistent
    feature → UnknownConstraintFeature (REQ-259).

Bug surfaced → REQ-269

The shared-child coverage revealed a real latent bug: a feature reachable
through two parents (a diamond / DAG) is mis-reported as a cycle by
validate_tree's BFS (global-visited-as-cycle). Confirmed failing with
cycle detected involving feature shared``. Filed as REQ-269 (v0.29.0)
with a regression test diamond_shared_child_is_not_a_cycle (`#[ignore]`d
until the fix lands, then flip to enforced).

Verification

cargo test -p rivet-core --test proptest_feature_model → 12 passed, 1
ignored. clippy --all-targets (1.97.0) clean. Test-only + artifact
change — the composition core and wasm seam are untouched.

Implements: REQ-266 · Verifies: REQ-257, REQ-259 · Refs: REQ-269, DD-076

🤖 Generated with Claude Code

The proptest strategies hard-coded `constraints: vec![]`, so the solver's
highest-risk code — cross-tree constraint evaluation, `implies`
propagation, `excludes`, and REQ-257's fail-loud gate — had ZERO
property-test coverage. Add:

- A constraint-generating strategy (implies/excludes/and/or/not over real
  feature-name leaves) kept as a typed value so the property can evaluate
  each constraint independently — an oracle to grade the solver against.
- prop_ok_means_constraints_actually_hold: if solve() returns Ok, every
  constraint genuinely holds under the resolved selection. This is the
  anti-silent-pass property — a return to the pre-REQ-257 `_ => true`
  blind spot would surface here as an Ok whose selection violates a
  constraint.
- prop_implies_propagation + prop_solver_never_panics_with_constraints.
- Deterministic fail-loud tests: an unevaluatable attribute comparison
  yields UnevaluatableConstraint (REQ-257); a constraint naming a
  nonexistent feature yields UnknownConstraintFeature (REQ-259).

Also surfaces a real latent bug: a shared child reachable through two
parents (a diamond / DAG) is mis-reported as a cycle by validate_tree's
BFS (global-visited-as-cycle). Filed as REQ-269 with a regression test
(diamond_shared_child_is_not_a_cycle, #[ignore]d until the fix); confirmed
failing with "cycle detected involving feature `shared`".

Confirmed with `cargo test -p rivet-core --test proptest_feature_model`
(12 passed, 1 ignored) and clippy --all-targets clean. Test-only + artifact
change; the wasm seam is untouched.

Implements: REQ-266
Verifies: REQ-257, REQ-259
Refs: REQ-269, DD-076
@github-actions

Copy link
Copy Markdown

📐 Rivet artifact delta

Change Count
Added 1
Removed 0
Modified 1
Downstream impacted (depth ≤ 5) 0

Graph

graph LR
  REQ_266["REQ-266"]:::modified
  REQ_269["REQ-269"]:::added
  classDef added fill:#d4edda,stroke:#28a745,color:#155724
  classDef removed fill:#f8d7da,stroke:#dc3545,color:#721c24
  classDef modified fill:#fff3cd,stroke:#ffc107,color:#856404
  classDef overflow fill:#e2e3e5,stroke:#6c757d,color:#495057,stroke-dasharray: 3 3
Loading
Added
  • REQ-269
Modified
ID Changes
REQ-266

📎 Full HTML dashboard attached as workflow artifact rivet-delta-pr-715download from the workflow run.

Posted by rivet-delta workflow. The graph shows only changed artifacts; open the HTML dashboard (above) for full context.

@codecov

codecov Bot commented Jul 16, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Performance Alert ⚠️

Possible performance regression was detected for benchmark 'Rivet Criterion Benchmarks'.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 1.20.

Benchmark suite Current: 6480063 Previous: 8abbb62 Ratio
store_insert/100 86586 ns/iter (± 913) 69126 ns/iter (± 437) 1.25
store_insert/1000 928388 ns/iter (± 9279) 740783 ns/iter (± 5800) 1.25
store_lookup/100 1947 ns/iter (± 8) 1522 ns/iter (± 25) 1.28
store_lookup/1000 25050 ns/iter (± 173) 18584 ns/iter (± 46) 1.35
store_lookup/10000 361448 ns/iter (± 9453) 265291 ns/iter (± 954) 1.36
store_by_type/100 96 ns/iter (± 0) 74 ns/iter (± 0) 1.30
store_by_type/1000 95 ns/iter (± 0) 74 ns/iter (± 0) 1.28
store_by_type/10000 95 ns/iter (± 0) 74 ns/iter (± 0) 1.28
schema_load_and_merge 1525307 ns/iter (± 29137) 1181005 ns/iter (± 22486) 1.29
link_graph_build/100 166807 ns/iter (± 1805) 126830 ns/iter (± 655) 1.32
link_graph_build/1000 1985681 ns/iter (± 5978) 1495979 ns/iter (± 9916) 1.33
validate/100 456973 ns/iter (± 6255) 341875 ns/iter (± 3801) 1.34
validate/1000 15016997 ns/iter (± 104351) 11276807 ns/iter (± 269977) 1.33
validate/10000 1101243769 ns/iter (± 20667852) 762756012 ns/iter (± 8858190) 1.44
traceability_matrix/100 4274 ns/iter (± 331) 3215 ns/iter (± 15) 1.33
traceability_matrix/1000 45133 ns/iter (± 271) 34302 ns/iter (± 168) 1.32
traceability_matrix/10000 734981 ns/iter (± 9516) 555669 ns/iter (± 4285) 1.32
diff/100 64047 ns/iter (± 197) 46183 ns/iter (± 131) 1.39
diff/1000 722526 ns/iter (± 13596) 521705 ns/iter (± 2750) 1.38
diff/10000 8238501 ns/iter (± 141364) 6163298 ns/iter (± 202417) 1.34
query/100 1120 ns/iter (± 2) 904 ns/iter (± 2) 1.24
query/1000 15042 ns/iter (± 35) 11284 ns/iter (± 99) 1.33
query/10000 225366 ns/iter (± 3195) 172570 ns/iter (± 970) 1.31
document_parse/10 21752 ns/iter (± 97) 17012 ns/iter (± 124) 1.28
document_parse/100 153972 ns/iter (± 1044) 117738 ns/iter (± 295) 1.31
document_parse/1000 1393310 ns/iter (± 22561) 1102249 ns/iter (± 26378) 1.26

This comment was automatically generated by workflow using github-action-benchmark.

@avrabe
avrabe merged commit ab9b24c into main Jul 16, 2026
30 checks passed
@avrabe
avrabe deleted the feat/req-266-constraint-proptests branch July 16, 2026 11:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant