|
| 1 | += Architecture Decision Record: 0005-lineage-acyclicity |
| 2 | +<!-- SPDX-License-Identifier: PMPL-1.0-or-later --> |
| 3 | +<!-- Copyright (c) 2026 Jonathan D.A. Jewell (hyperpolymath) <j.d.a.jewell@open.ac.uk> --> |
| 4 | + |
| 5 | +# 5. Lineage is enforced as a DAG at write time |
| 6 | + |
| 7 | +Date: 2026-05-14 |
| 8 | + |
| 9 | +## Status |
| 10 | + |
| 11 | +Accepted |
| 12 | + |
| 13 | +## Context |
| 14 | + |
| 15 | +`README.adoc` and `src/abi/mod.rs::LineageEdge` describe the |
| 16 | +lineage representation as a **DAG** of data derivations. Until |
| 17 | +V-L2-I1 (#42) closed self-loops at the storage layer, *nothing* |
| 18 | +in the system enforced that claim. After #42, a single-hop |
| 19 | +self-loop is rejected by a CHECK constraint, but multi-hop |
| 20 | +cycles (A → B → A, A → B → C → A, …) still slip through both |
| 21 | +the SQL DDL and the application code. |
| 22 | + |
| 23 | +The lineage table just stores edges; cycles only matter if a |
| 24 | +consumer of the table tries to traverse them. Today no consumer |
| 25 | +does — the lineage graph is write-only — so the cost of a cycle |
| 26 | +is invisible. But every traversal we plan (V-L1-D2 lineage |
| 27 | +queries, V-L1-D3 impact-analysis, the upcoming `verisimiser |
| 28 | +provenance --transitive` mode) eventually needs an acyclicity |
| 29 | +invariant or it will hang / produce wrong answers. |
| 30 | + |
| 31 | +Two coherent choices exist: |
| 32 | + |
| 33 | +1. **Enforce DAG** — refuse inserts that would create a cycle. |
| 34 | + Pay the write-time cost (a per-insert reachability check); get |
| 35 | + correctness-by-construction for every future traversal. |
| 36 | +2. **Rename to "Lineage Graph"** — drop the "DAG" claim; document |
| 37 | + that consumers must defend against cycles themselves. Cheaper |
| 38 | + writes; every traversal pays the cost forever. |
| 39 | +
|
| 40 | +## Decision |
| 41 | + |
| 42 | +**Enforce DAG at the application write path.** Reject any insert |
| 43 | +where `(source, target)` would close a cycle in |
| 44 | +`verisimdb_lineage_graph`. |
| 45 | + |
| 46 | +Implementation outline (for the V-L1-G2 follow-up): |
| 47 | + |
| 48 | +* Before inserting an edge `(s, t, …)`, run a reachability check |
| 49 | + from `t` to `s` over existing edges. If `t` can reach `s`, the |
| 50 | + new edge would close a cycle; refuse with a typed error |
| 51 | + carrying the offending path. |
| 52 | +* Reachability is a transitive-closure query. SQLite ≥ 3.8.3 and |
| 53 | + PostgreSQL both support recursive CTEs; verisimiser emits the |
| 54 | + query at codegen time so the cost stays in SQL. |
| 55 | +* Concurrent inserts must serialise on the cycle check or the |
| 56 | + invariant can be lost between read and write. V-L2-L1 (#31) |
| 57 | + already constrains chain forks via per-entity serialisation; |
| 58 | + this ADR extends the same pattern to the lineage table. |
| 59 | +
|
| 60 | +Storage layer keeps the existing single-hop CHECK (#42) as the |
| 61 | +last line of defence in case the application path is bypassed. |
| 62 | + |
| 63 | +### Alternative considered |
| 64 | + |
| 65 | +Rename to "Lineage Graph" and ship cycles. Rejected because: |
| 66 | + |
| 67 | +* Every documented use of lineage (where-from, what-affects-this, |
| 68 | + impact-analysis) needs acyclicity for soundness. Pushing the |
| 69 | + burden to consumers means every future query is harder to |
| 70 | + write and easier to get wrong. |
| 71 | +* The write-time cost of a recursive CTE on the typical lineage |
| 72 | + table (sparse, low fan-out per entity) is negligible. The |
| 73 | + read-time cost of defending against cycles in every consumer is |
| 74 | + not. |
| 75 | +* The "DAG" claim is already public (README); silently weakening |
| 76 | + it to "Graph" is a downgrade users would notice. |
| 77 | +
|
| 78 | +### Alternative considered (also rejected) |
| 79 | + |
| 80 | +Enforce in SQL only via triggers. Rejected because: |
| 81 | + |
| 82 | +* Triggers do not work uniformly across SQLite/PostgreSQL/ |
| 83 | + MongoDB; the codegen layer would have to ship three flavours. |
| 84 | +* Application-level enforcement gives a structured error type |
| 85 | + carrying the cycle path, useful for tooling. Triggers can't |
| 86 | + easily return that. |
| 87 | +
|
| 88 | +## Consequences |
| 89 | + |
| 90 | +### Positive |
| 91 | + |
| 92 | +* Every future traversal can assume acyclicity. No defensive |
| 93 | + cycle-breaking in consumers; queries are simpler and faster. |
| 94 | +* The README's "DAG" claim becomes truthful. |
| 95 | +* `verisimiser doctor` can add a `lineage-acyclic` check that |
| 96 | + surveys existing rows and confirms the invariant holds. |
| 97 | +
|
| 98 | +### Negative |
| 99 | + |
| 100 | +* Writes are O(reachable subgraph from target) rather than O(1). |
| 101 | + Pathological wide graphs make writes slower. Sparse lineage |
| 102 | + is the expected case, but we should monitor. |
| 103 | +* Concurrent inserts that race to close the same cycle: one |
| 104 | + succeeds, one fails. Need a clear error message. |
| 105 | +
|
| 106 | +### Neutral |
| 107 | + |
| 108 | +* The single-hop CHECK (#42) still fires for the simplest case |
| 109 | + and gives a fast-path error. |
| 110 | +
|
| 111 | +## Cross-references |
| 112 | + |
| 113 | +* V-L2-I1 (#42) — single-hop self-reference CHECK (already |
| 114 | + merged). |
| 115 | +* V-L1-G2 (to be filed) — implementation of the multi-hop |
| 116 | + cycle check. |
| 117 | +* V-L2-L1 (#31) — per-entity write serialisation; same pattern |
| 118 | + applies here. |
| 119 | +* `src/codegen/overlay.rs::generate_lineage_table` — where the |
| 120 | + storage-layer single-hop CHECK lives. |
| 121 | +* `src/abi/mod.rs::LineageEdge` — where the application-layer |
| 122 | + cycle check will be added. |
0 commit comments