Skip to content

Commit 598998a

Browse files
docs(adr): 0005-lineage-acyclicity — enforce DAG at write time (#93)
Closes #65. The README and `LineageEdge` doc both claim "DAG". Until #42 closed single-hop self-loops at the storage layer, nothing enforced that claim. After #42, multi-hop cycles (A → B → A, …) still slip through. Lineage is write-only today, so the cost of a cycle is invisible — but every documented traversal (V-L1-D2 lineage queries, impact-analysis, transitive provenance) needs acyclicity for soundness. ADR-0005 picks **enforce-at-write-time**: * Refuse inserts where `(source, target)` would close a cycle. * Implementation uses recursive CTEs (works on SQLite 3.8.3+ and Postgres) emitted at codegen time, so the cost stays in SQL. * Concurrent inserts serialise on the cycle check (same V-L2-L1 per-entity pattern). * Storage-layer single-hop CHECK (#42) remains as the last line of defence. Alternatives considered & rejected: - Rename to "Lineage Graph" and ship cycles. Rejected: every documented use of lineage needs acyclicity for soundness; write cost is small, read cost across every future consumer is not. - SQL triggers only. Rejected: triggers vary across backends; application-layer enforcement returns a structured error type carrying the cycle path. The implementation lands in V-L1-G2 (to be filed). This ADR is binding for that follow-up. Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
1 parent 4749646 commit 598998a

1 file changed

Lines changed: 122 additions & 0 deletions

File tree

Lines changed: 122 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,122 @@
1+
= Architecture Decision Record: 0005-lineage-acyclicity
2+
<!-- SPDX-License-Identifier: PMPL-1.0-or-later -->
3+
<!-- Copyright (c) 2026 Jonathan D.A. Jewell (hyperpolymath) <j.d.a.jewell@open.ac.uk> -->
4+
5+
# 5. Lineage is enforced as a DAG at write time
6+
7+
Date: 2026-05-14
8+
9+
## Status
10+
11+
Accepted
12+
13+
## Context
14+
15+
`README.adoc` and `src/abi/mod.rs::LineageEdge` describe the
16+
lineage representation as a **DAG** of data derivations. Until
17+
V-L2-I1 (#42) closed self-loops at the storage layer, *nothing*
18+
in the system enforced that claim. After #42, a single-hop
19+
self-loop is rejected by a CHECK constraint, but multi-hop
20+
cycles (A → B → A, A → B → C → A, …) still slip through both
21+
the SQL DDL and the application code.
22+
23+
The lineage table just stores edges; cycles only matter if a
24+
consumer of the table tries to traverse them. Today no consumer
25+
does — the lineage graph is write-only — so the cost of a cycle
26+
is invisible. But every traversal we plan (V-L1-D2 lineage
27+
queries, V-L1-D3 impact-analysis, the upcoming `verisimiser
28+
provenance --transitive` mode) eventually needs an acyclicity
29+
invariant or it will hang / produce wrong answers.
30+
31+
Two coherent choices exist:
32+
33+
1. **Enforce DAG** — refuse inserts that would create a cycle.
34+
Pay the write-time cost (a per-insert reachability check); get
35+
correctness-by-construction for every future traversal.
36+
2. **Rename to "Lineage Graph"** — drop the "DAG" claim; document
37+
that consumers must defend against cycles themselves. Cheaper
38+
writes; every traversal pays the cost forever.
39+
40+
## Decision
41+
42+
**Enforce DAG at the application write path.** Reject any insert
43+
where `(source, target)` would close a cycle in
44+
`verisimdb_lineage_graph`.
45+
46+
Implementation outline (for the V-L1-G2 follow-up):
47+
48+
* Before inserting an edge `(s, t, …)`, run a reachability check
49+
from `t` to `s` over existing edges. If `t` can reach `s`, the
50+
new edge would close a cycle; refuse with a typed error
51+
carrying the offending path.
52+
* Reachability is a transitive-closure query. SQLite ≥ 3.8.3 and
53+
PostgreSQL both support recursive CTEs; verisimiser emits the
54+
query at codegen time so the cost stays in SQL.
55+
* Concurrent inserts must serialise on the cycle check or the
56+
invariant can be lost between read and write. V-L2-L1 (#31)
57+
already constrains chain forks via per-entity serialisation;
58+
this ADR extends the same pattern to the lineage table.
59+
60+
Storage layer keeps the existing single-hop CHECK (#42) as the
61+
last line of defence in case the application path is bypassed.
62+
63+
### Alternative considered
64+
65+
Rename to "Lineage Graph" and ship cycles. Rejected because:
66+
67+
* Every documented use of lineage (where-from, what-affects-this,
68+
impact-analysis) needs acyclicity for soundness. Pushing the
69+
burden to consumers means every future query is harder to
70+
write and easier to get wrong.
71+
* The write-time cost of a recursive CTE on the typical lineage
72+
table (sparse, low fan-out per entity) is negligible. The
73+
read-time cost of defending against cycles in every consumer is
74+
not.
75+
* The "DAG" claim is already public (README); silently weakening
76+
it to "Graph" is a downgrade users would notice.
77+
78+
### Alternative considered (also rejected)
79+
80+
Enforce in SQL only via triggers. Rejected because:
81+
82+
* Triggers do not work uniformly across SQLite/PostgreSQL/
83+
MongoDB; the codegen layer would have to ship three flavours.
84+
* Application-level enforcement gives a structured error type
85+
carrying the cycle path, useful for tooling. Triggers can't
86+
easily return that.
87+
88+
## Consequences
89+
90+
### Positive
91+
92+
* Every future traversal can assume acyclicity. No defensive
93+
cycle-breaking in consumers; queries are simpler and faster.
94+
* The README's "DAG" claim becomes truthful.
95+
* `verisimiser doctor` can add a `lineage-acyclic` check that
96+
surveys existing rows and confirms the invariant holds.
97+
98+
### Negative
99+
100+
* Writes are O(reachable subgraph from target) rather than O(1).
101+
Pathological wide graphs make writes slower. Sparse lineage
102+
is the expected case, but we should monitor.
103+
* Concurrent inserts that race to close the same cycle: one
104+
succeeds, one fails. Need a clear error message.
105+
106+
### Neutral
107+
108+
* The single-hop CHECK (#42) still fires for the simplest case
109+
and gives a fast-path error.
110+
111+
## Cross-references
112+
113+
* V-L2-I1 (#42) — single-hop self-reference CHECK (already
114+
merged).
115+
* V-L1-G2 (to be filed) — implementation of the multi-hop
116+
cycle check.
117+
* V-L2-L1 (#31) — per-entity write serialisation; same pattern
118+
applies here.
119+
* `src/codegen/overlay.rs::generate_lineage_table` — where the
120+
storage-layer single-hop CHECK lives.
121+
* `src/abi/mod.rs::LineageEdge` — where the application-layer
122+
cycle check will be added.

0 commit comments

Comments
 (0)