|
| 1 | +# ADR-0071: Dataset semantic-layer depth — multi-hop joins and matrix pivoting |
| 2 | + |
| 3 | +## Status |
| 4 | + |
| 5 | +Proposed |
| 6 | + |
| 7 | +## Context |
| 8 | + |
| 9 | +ADR-0021 established the `dataset` semantic layer: a base `object`, relationships |
| 10 | +to `include` (joins **derived** from the object graph — no hand-written `ON` |
| 11 | +clauses), and named `dimensions` / `measures` that presentations bind by name. |
| 12 | +RLS/tenant scoping is enforced per joined object at runtime (D-C, fail-closed). |
| 13 | + |
| 14 | +Two capability gaps remain, both flagged as follow-ups in ADR-0021: |
| 15 | + |
| 16 | +1. **Single-hop joins only.** `Dataset.include` is `string[]` of relationship |
| 17 | + **names** (lookup / master_detail fields on the *base* object), and a |
| 18 | + dimension/measure `field` uses a one-dot `relationship.field` path. |
| 19 | + `native-sql-strategy.qualifyAndRegisterJoin` splits on the **first** dot: |
| 20 | + `account.region` → `LEFT JOIN account` + column `"account"."region"`. A |
| 21 | + two-hop path like `account.owner.region` is parsed as alias=`account`, |
| 22 | + column=`owner.region` → invalid SQL. You cannot group/aggregate by a field |
| 23 | + two relationships away (e.g. `opportunity → account → owner → region`). |
| 24 | + |
| 25 | +2. **Matrix / "across" is flattened, not pivoted.** `Report` already declares |
| 26 | + `rows` (down) and `columns` (across) for `type: 'matrix'`, and the dataset |
| 27 | + query can group by both. But the dataset report renderer shows a **flat** |
| 28 | + table (rows ∪ columns as combined groupings) — it does not pivot the |
| 29 | + `columns` dimension into a 2-D grid (ADR-0021 D2 follow-up). A true matrix |
| 30 | + (rows × distinct-column-values × measure cells) isn't rendered. |
| 31 | + |
| 32 | +## Prior art / industry alignment |
| 33 | + |
| 34 | +- **Curated join graph, not arbitrary SQL.** Salesforce *report types* pre-wire |
| 35 | + a primary object + up to 3 related objects; Looker *Explores* declare the |
| 36 | + joins a developer blesses; Power BI / Tableau model **relationships** and the |
| 37 | + engine resolves the path; Cube (this platform's runtime) resolves **transitive |
| 38 | + joins** from pairwise relationship declarations. All keep joins governed and |
| 39 | + derived from the object graph — exactly ADR-0021's D-C stance. This ADR stays |
| 40 | + in that lane (no `ON` clauses). |
| 41 | +- **Spanning depth.** Salesforce report types reach 4 objects (≈3 to-one hops); |
| 42 | + SOQL allows 5 levels of parent traversal. A small, fixed depth covers the vast |
| 43 | + majority of real reports. |
| 44 | +- **The fan-out lesson.** Looker (*symmetric aggregates*) and Tableau (*relationships |
| 45 | + vs. joins*) both exist because joining across a **to-many** edge and then |
| 46 | + aggregating **double-counts**. The robust answer for a governed, AI-authored, |
| 47 | + RLS-scoped layer is to **not** traverse to-many in the join chain at all (see |
| 48 | + Decision A) rather than to build symmetric-aggregate machinery now. |
| 49 | +- **Matrix.** Salesforce matrix reports allow ≤2 row groupings + ≤2 column |
| 50 | + groupings; Power BI's matrix visual pivots with overflow handling. P1 targets |
| 51 | + the single-column-grouping MVP with a display-side overflow cap. |
| 52 | + |
| 53 | +## Decision |
| 54 | + |
| 55 | +### A. Multi-hop joins (framework: spec + compiler + strategy + RLS) |
| 56 | + |
| 57 | +- **To-one only (the correctness boundary).** Multi-hop traversal is restricted |
| 58 | + to **to-one** relationships — `lookup` / `master_detail` (child→parent), which |
| 59 | + is exactly what `include` holds today. To-one chains **never fan out**, so |
| 60 | + existing `SUM`/`COUNT`/etc. stay correct with **zero** symmetric-aggregate |
| 61 | + machinery. Traversing a **to-many** (child) relationship inside a dataset's |
| 62 | + join chain is explicitly **out of scope** (it needs symmetric aggregates / |
| 63 | + sub-query rollups — a separate feature). Consequently the depth cap below is a |
| 64 | + **performance/complexity** guard, not a correctness one. |
| 65 | +- **Spec.** `Dataset.include` accepts relationship **paths** — dotted chains of |
| 66 | + to-one relationship field names from the base object (`'account'`, |
| 67 | + `'account.owner'`). A dimension/measure `field` may reference any field |
| 68 | + reachable along a **declared** path (`account.owner.region`). The dotted path |
| 69 | + **is** the join alias, so paths are **self-disambiguating** (no named-join |
| 70 | + disambiguation like Power BI active/inactive or Looker aliased joins). A depth |
| 71 | + cap (default **3 hops** → 4 objects, Salesforce-report-type parity) bounds join |
| 72 | + count/perf; undeclared paths are still rejected (D-C unchanged). |
| 73 | +- **Compiler** (`dataset-compiler`). Expand each `include` path into the ordered |
| 74 | + join chain; `cube.joins` is keyed by the full path alias (`account.owner`) and |
| 75 | + carries `{ parentAlias, fkField, targetObject }` so the chain is reconstructable. |
| 76 | +- **Strategy** (`native-sql-strategy.qualifyAndRegisterJoin`). For `a.b.c`, |
| 77 | + register the chain (`base → a` on alias `a`; `a → b` on alias `a.b`) and |
| 78 | + qualify the column as `"a.b"."c"`. Emit `LEFT JOIN`s (outer — base rows |
| 79 | + without a related record still appear, the report-friendly default) in |
| 80 | + dependency order. Base columns stay qualified with the base table so shared |
| 81 | + column names remain unambiguous across all hops. |
| 82 | +- **RLS (D-C, fail-closed).** Apply the tenant read-scope to **every** object in |
| 83 | + the chain, not just the first hop. The strategy already scopes per joined |
| 84 | + alias; generalize the loop to each hop's target object. |
| 85 | +- **UI** (objectui). `useDatasetFieldCatalog` lazily expands a relationship's own |
| 86 | + **to-one** relationships (one more level on demand) so the field picker offers |
| 87 | + `account.owner.region`; the include editor shows/edits paths; the existing |
| 88 | + `missingRelationship` author-time validation generalizes to paths. |
| 89 | + |
| 90 | +### B. Matrix pivot (mostly objectui renderer) |
| 91 | + |
| 92 | +- **Query unchanged.** Group by `rows ∪ columns` — the across dimension is just |
| 93 | + another `groupBy`. |
| 94 | +- **Renderer** (`DatasetReportRenderer`). When `type === 'matrix'` and `columns` |
| 95 | + is non-empty, **pivot** the flat result into a 2-D grid: distinct `columns` |
| 96 | + values become column headers, `rows` go down the side, measure(s) fill the |
| 97 | + cells. Cap distinct column values at a **display** ceiling (default **50**, |
| 98 | + with a "+N more — refine with a filter" notice) — a render-side guard, not a |
| 99 | + query limit. MVP = a single column grouping; ≤2 row/col groupings (Salesforce |
| 100 | + parity) is a fast-follow. |
| 101 | +- **Spec unchanged** (`rows` / `columns` already exist on `Report`). |
| 102 | + |
| 103 | +## Consequences |
| 104 | + |
| 105 | +- New capability: analytics two-plus to-one hops deep, and true matrix reports. |
| 106 | +- Cost/risk: multi-hop joins multiply `LEFT JOIN`s (perf) and widen the RLS |
| 107 | + surface (each hop scoped) — bounded by the depth cap + declared-path allowlist. |
| 108 | + No fan-out risk by construction (to-one only). Matrix column cardinality is |
| 109 | + bounded by the display cap. |
| 110 | +- **Backward compatible.** Single-hop `include` and flat rows keep working |
| 111 | + unchanged; the depth cap and "flat unless `matrix` + `columns`" defaults |
| 112 | + preserve current behaviour. |
| 113 | + |
| 114 | +## Phasing |
| 115 | + |
| 116 | +- **P1 — matrix pivot (objectui-only, lowest risk).** Render the matrix grid from |
| 117 | + the existing `rows`/`columns`. No spec/query change. Ships independently. |
| 118 | +- **P2 — multi-hop (framework).** Spec path support + compiler chain + strategy |
| 119 | + chained joins + per-hop RLS, behind the depth cap, **to-one only**. Gated by |
| 120 | + the ADR-0021 reconciliation harness (old-vs-new numbers). |
| 121 | +- **P3 — objectui catalog/UI.** Multi-hop field picker + include-path editor + |
| 122 | + path-aware author-time validation. |
| 123 | + |
| 124 | +## Alternatives considered |
| 125 | + |
| 126 | +- **Arbitrary join predicates / raw SQL** — rejected (ADR-0021 D-C: joins are |
| 127 | + derived from the object graph, no `ON` clauses; keeps datasets reviewable and |
| 128 | + RLS-safe). Matches Salesforce/Cube/Power BI, not the developer-only Looker |
| 129 | + `sql_on`. |
| 130 | +- **To-many traversal with symmetric aggregates** — deferred. Correct |
| 131 | + cross-to-many aggregation (Looker-style symmetric aggregates) is a separate, |
| 132 | + larger feature; the to-one boundary delivers the common case safely first. |
| 133 | +- **Edge-graph + automatic path resolution** (declare relationship edges, let the |
| 134 | + engine find the path — Cube/Power BI style) — rejected for v1 in favour of |
| 135 | + explicit dotted paths: smaller delta, self-disambiguating, and per-path |
| 136 | + RLS-auditable. |
| 137 | +- **Materialized / precomputed pivots** — out of scope; query-time pivot is |
| 138 | + sufficient at expected cardinalities (bounded by the display cap). |
| 139 | + |
| 140 | +## Related |
| 141 | + |
| 142 | +- ADR-0021 (analytics dataset semantic layer) — the foundation; D2 / join |
| 143 | + follow-ups this ADR addresses. |
0 commit comments