Skip to content

Commit 99abd2a

Browse files
committed
docs(dst): record the PR-E cross-backend walk + omnigraph-dst crate
MATRIX.md: D5 CLI backend ❌ deferred → ✅ (PR-E); split out the still-deferred long-lived server walk; update the roadmap summary. testing.md: list cli_cross_backend_walk in the CLI suites, add its run command, and document the omnigraph-dst crate (the Backend trait + Embedded/Cli, the dev-dep cycle, and why the white-box battery stays embedded-only).
1 parent 62e9104 commit 99abd2a

2 files changed

Lines changed: 16 additions & 8 deletions

File tree

crates/omnigraph/tests/dst/MATRIX.md

Lines changed: 11 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -75,8 +75,9 @@ Legend: ✅ sampled · 🟡 partial · ❌ unsampled · ⏸️ deferred-by-plan
7575
### D5 — context *(the dimension with the biggest blind spots)*
7676
| context | status | note |
7777
|---------|--------|------|
78-
| embedded backend || the only backend |
79-
| **CLI / long-lived server backend** | ❌ deferred | crate-boundary friction: harness is in `omnigraph-engine` tests, the `omnigraph` binary is in `omnigraph-cli` (`CARGO_BIN_EXE_omnigraph` is cross-package, unavailable), and the natural home — `omnigraph-cli` tests — can't reach the harness modules. Plus subprocess-per-op is slow/flaky. Needs the harness op/model extracted to a shared crate first. |
78+
| embedded backend || drives `Embedded` (the in-process `Backend` impl) + the white-box battery |
79+
| **CLI backend (subprocess)** | ✅ (PR-E) | `cli_cross_backend_walk` (in `omnigraph-cli` tests) — the SAME seeded walk drives `Embedded` AND `Cli` (`omnigraph-dst::Cli`, binary via `CARGO_BIN_EXE_omnigraph`), asserting per-step black-box agreement (slug sets + edge count). Unblocked by extracting op/model/invariants/fault behind the `Backend` trait into the shared `omnigraph-dst` crate. White-box battery stays embedded-only by construction; the CLI arm runs the black-box oracles. One known contract difference allow-listed: `repair --confirm` (no `--force`) exits non-zero refusing suspicious drift where embedded `repair(force=false)` returns Ok. |
80+
| **long-lived server backend** | ❌ deferred | `parity_matrix.rs` covers single-op CLI-vs-server; a server-arm generative walk is the remaining D5 gap. |
8081
| local FS || |
8182
| **S3 (RustFS/MinIO)** | ✅ (PR-D) | `s3_battery_holds` — full battery on `s3://`, env-gated (`OMNIGRAPH_S3_TEST_BUCKET`) |
8283
| **parser/loader fuzz** | ✅ (PR-D) | `fuzz::*` — proptest dup/malformed-injecting; `cargo-fuzz`/libFuzzer deferred (needs nightly) |
@@ -131,10 +132,14 @@ seam cannot induce a `RolledPastExpected` sidecar.
131132
the walk *discovers* sidecar/CAS bugs instead of the cells being scripted. ❌
132133
6. **determinism/replay-equality** (PR-C) — ✅ DONE (`--features dst` seam +
133134
`replay_equality_same_seed`). **S3 context + parser/loader fuzz** (PR-D) — ✅ DONE
134-
(`s3_battery_holds`, `fuzz::*`). **Still ⏸️/❌:** CLI/server backends (deferred,
135-
crate-boundary — see D5 table), `cargo-fuzz`/libFuzzer (needs nightly),
136-
`porcupine` linearizability, vector/FTS/rrf read shapes, wrapping the main
137-
walks in `with_seed`.
135+
(`s3_battery_holds`, `fuzz::*`). **CLI cross-backend walk** (PR-E) — ✅ DONE: the
136+
harness op/model/invariants/fault extracted behind a `Backend` trait into the
137+
shared `omnigraph-dst` crate, so the same seeded walk runs embedded AND via the
138+
CLI subprocess (`cli_cross_backend_walk`, see D5 table). **Still ⏸️/❌:**
139+
long-lived *server* backend walk (deferred — `parity_matrix` covers single-op
140+
CLI-vs-server), `cargo-fuzz`/libFuzzer (needs nightly), `porcupine`
141+
linearizability, vector/FTS/rrf read shapes, wrapping the main walks in
142+
`with_seed`.
138143

139144
## The standing rule
140145
When a bug is found *outside* this harness, before closing it: add its row to the

docs/dev/testing.md

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ This file is the always-on map of the test surface. **Consult it before every ta
77
| Crate | Path | Style |
88
|---|---|---|
99
| `omnigraph` (engine) | `crates/omnigraph/tests/` | Integration tests (28 files), fixture-driven, share `tests/helpers/mod.rs` |
10-
| `omnigraph-cli` | `crates/omnigraph-cli/tests/` | Per-area suites (post-modularization): `cli_cluster.rs` (cluster command surface + operator-actor cascade), `cli_cluster_e2e.rs` (spawned-binary lifecycle compositions — lost-state re-import recovery, out-of-band drift, graph-root destruction, multi-graph mixed-disposition convergence), `cli_data.rs` (load/read/change/branch/commit/export/snapshot/policy/embed/maintenance + operator format cascade), `cli_schema_config.rs` (init/config, schema plan/apply), `cli_queries.rs`, `parity_matrix.rs` (RFC-009 Phase 1: the embedded-vs-remote referee — every forked verb run against both arms with matched Cedar policy and the same actor, scrubbed-JSON + exit-code equality; divergences are pinned in its `KNOWN_DIVERGENCES` ledger, never silently repaired), `cli_dst_parity.rs` (DST cross-backend smoke — a DST-flavored op SEQUENCE [init → 2× `load --merge` → insert → edge-free delete → query] run against the **embedded SDK** and the **CLI subprocess** on twin local graphs must agree on the final slug set; fills the gap parity_matrix leaves — the embedded arm + multi-op sequences; reuses `support::cli()`), `system_local.rs` (full-cycle cluster lifecycle with a spawned `--cluster` server, applied-policy enforcement over HTTP, keyed-credential auth, operator aliases), `system_remote.rs`; share `tests/support/mod.rs` (hermetic `OMNIGRAPH_HOME` by default) |
10+
| `omnigraph-cli` | `crates/omnigraph-cli/tests/` | Per-area suites (post-modularization): `cli_cluster.rs` (cluster command surface + operator-actor cascade), `cli_cluster_e2e.rs` (spawned-binary lifecycle compositions — lost-state re-import recovery, out-of-band drift, graph-root destruction, multi-graph mixed-disposition convergence), `cli_data.rs` (load/read/change/branch/commit/export/snapshot/policy/embed/maintenance + operator format cascade), `cli_schema_config.rs` (init/config, schema plan/apply), `cli_queries.rs`, `parity_matrix.rs` (RFC-009 Phase 1: the embedded-vs-remote referee — every forked verb run against both arms with matched Cedar policy and the same actor, scrubbed-JSON + exit-code equality; divergences are pinned in its `KNOWN_DIVERGENCES` ledger, never silently repaired), `cli_dst_parity.rs` (DST cross-backend smoke — a DST-flavored op SEQUENCE [init → 2× `load --merge` → insert → edge-free delete → query] run against the **embedded SDK** and the **CLI subprocess** on twin local graphs must agree on the final slug set; fills the gap parity_matrix leaves — the embedded arm + multi-op sequences; reuses `support::cli()`), `cli_cross_backend_walk.rs` (PR-E — the FULL generative DST walk: the same seeded op stream from the shared `omnigraph-dst` crate drives the embedded `Omnigraph` SDK AND the `omnigraph` CLI subprocess in lockstep, asserting per-step black-box agreement on Person/Doc slug sets + traversable edge count + op-success parity; the white-box battery stays embedded-only, so the CLI arm runs the black-box oracles; one known contract difference allow-listed — `repair --confirm` exits non-zero on suspicious drift where embedded `repair(force=false)` returns Ok), `system_local.rs` (full-cycle cluster lifecycle with a spawned `--cluster` server, applied-policy enforcement over HTTP, keyed-credential auth, operator aliases), `system_remote.rs`; share `tests/support/mod.rs` (hermetic `OMNIGRAPH_HOME` by default) |
1111
| `omnigraph-cluster` | mostly in-source `#[cfg(test)] mod tests`; `tests/failpoints.rs` (feature-gated); `tests/s3_cluster.rs` (bucket-gated full lifecycle on object storage) | Cluster config parser, local JSON state diff, state CAS/lock handling/recovery, read-only validate/plan/status plus explicit refresh/import graph observations, config-only apply (content-addressed payload publish, disposition gating, composite-digest convergence, idempotent re-apply), catalog payload verification (status read-only, refresh drift + self-heal), failpoint crash-mid-apply / CAS-race coverage, Stage 4A graph creation (create executor, recovery sidecars + sweep rows, create crash windows), Stage 4B schema apply (migration previews in plan, schema executor, schema-apply sweep classification, schema crash windows), Stage 4C gated deletes (digest-bound approvals, delete executor + tombstones, delete sweep rows, delete crash windows), and 5A policy binding metadata (applies_to in the applied revision, binding-change diffing + convergence, pre-5A backfill), and the 5B serving-snapshot read API (converged read, refusal rows) |
1212
| `omnigraph-server` | `crates/omnigraph-server/tests/` | Per-area suites (post-modularization): `auth_policy.rs`, `data_routes.rs`, `schema_routes.rs`, `stored_queries.rs`, `multi_graph.rs` (cluster-mode boot — converged serving, policy binding wiring, boot refusals — + the concurrent branch-ops matrix), `boot_settings.rs` (mode inference, PolicySource), `s3.rs` (bucket-gated: single-graph serving + config-free `--cluster s3://` boot), `openapi.rs` (OpenAPI drift / regeneration); share `tests/support/mod.rs` |
1313
| `omnigraph-compiler` | mostly in-source `#[cfg(test)] mod tests` | Parser, type-checker, IR lowering, lint |
@@ -81,10 +81,13 @@ The DST harness (`dst.rs` + `dst/`, above) has several run modes; each is its ow
8181
cargo test -p omnigraph-engine --test dst # the generative walks + battery + regressions + fuzz (default; S3 cell skips)
8282
cargo test -p omnigraph-engine --features dst --test dst # + the replay-equality oracle (the `dst` determinism feature, below)
8383
cargo test -p omnigraph-engine --features failpoints --test dst_recovery # the recovery / #296-class cells
84-
cargo test -p omnigraph-cli --test cli_dst_parity # the embedded-vs-CLI cross-backend smoke
84+
cargo test -p omnigraph-cli --test cli_dst_parity # the embedded-vs-CLI cross-backend smoke (fixed sequence)
85+
cargo test -p omnigraph-cli --test cli_cross_backend_walk # the FULL generative embedded-vs-CLI walk (PR-E)
8586
OMNIGRAPH_S3_TEST_BUCKET=… cargo test -p omnigraph-engine --test dst s3_battery_holds # the S3-context battery
8687
```
8788

89+
- **The `omnigraph-dst` crate** (`crates/omnigraph-dst`, dev-only, NOT a workspace default-member) holds the reusable harness primitives — the op alphabet, reference `model`, white-box invariant battery, the `FaultAdapter`, and the `Backend` trait with its `Embedded` (in-process) and `Cli` (subprocess) impls — so the SAME seeded walk can run embedded (engine `dst.rs`) AND cross-backend (`cli_cross_backend_walk.rs`). It depends on `omnigraph-engine` and is itself a `[dev-dependencies]` of both the engine and the CLI: a dev-dependency cycle cargo supports (the lib builds without dev-deps). The white-box battery needs the real `Omnigraph` handle (via `Embedded::db()`), so it is embedded-only; a cross-backend walk runs the black-box oracles (count/content/edges==model via `Backend::query`).
90+
8891
- **The `dst` feature** (`dst = []` in `crates/omnigraph/Cargo.toml`) is a TEST-ONLY deterministic-replay seam, structured exactly like `failpoints` (zero production cost — the `#[cfg(feature="dst")]` blocks compile away). `src/dst.rs` exposes `next_ulid()`/`now_micros()` that the engine's observable ULID/timestamp sites call instead of `Ulid::new()`/`SystemTime::now()`; without the feature (or with no provider in scope via `dst::with_seed`) they fall back to the real source. The harness uses it for `replay_equality_same_seed` (same seed → identical engine id/clock fingerprint + outcome). Bit-identical replay is **engine-layer only** — Lance internals (compaction/index ids, threadpool scheduling) stay non-deterministic.
8992
- See `crates/omnigraph/tests/dst/MATRIX.md` for the coverage ledger (which D1–D5 cells are sampled, the hidden dimensions found by checking against externally-fixed bugs like #296, and what's deferred).
9093

0 commit comments

Comments
 (0)