diff --git a/.github/workflows/website.yml b/.github/workflows/website.yml index e4373c67..8da8bfd0 100644 --- a/.github/workflows/website.yml +++ b/.github/workflows/website.yml @@ -10,7 +10,9 @@ on: - 'scripts/publish-artifacts.py' - 'scripts/publish-resources.py' - 'scripts/check-website-links.py' + - 'scripts/helix_validate_artifact_meta.py' - 'tests/validate-website-generated.sh' + - 'workflows/activities/**/artifacts/**' - '.vale.ini' - '.vale/**' - '.github/workflows/website.yml' @@ -65,6 +67,9 @@ jobs: - name: Generator drift + coverage gate run: bash tests/validate-website-generated.sh + - name: Validate artifact-type schemas + run: uv run scripts/helix_validate_artifact_meta.py + - name: Build site (Hugo) working-directory: website # Build with the production base path so the link check validates the diff --git a/docs/helix/02-design/adr/ADR-004-dependencies-encoding.md b/docs/helix/02-design/adr/ADR-004-dependencies-encoding.md new file mode 100644 index 00000000..030ba2ae --- /dev/null +++ b/docs/helix/02-design/adr/ADR-004-dependencies-encoding.md @@ -0,0 +1,110 @@ +--- +ddx: + id: ADR-004 + depends_on: + - helix.prd +--- +# ADR-004: Artifact Dependencies Are Encoded in `meta.yml.relationships` Only (No Separate `dependencies.yaml`) + +| Date | Status | Deciders | Related | Confidence | +|------|--------|----------|---------|------------| +| 2026-05-30 | Accepted | HELIX maintainers | artifact-schema, frame action, audit plan 2026-05-30 | High | + +## Context + +| Aspect | Description | +|--------|-------------| +| Problem | The artifact-type rubric historically expected each artifact-type directory to ship a separate `dependencies.yaml` file describing inputs, outputs, and validation rules. In practice, no such file exists for any artifact type, while the same information is already encoded in `meta.yml` under `relationships` (and adjacent `validation.*` sections). The audit (`docs/helix/02-design/plan-2026-05-30-artifact-types-and-concerns-audit.md`) found `dependencies.yaml` absent for 48/48 artifact types. | +| Current State | `meta.yml.relationships` already names `depends_on` and `informs` per artifact type. `meta.yml.validation` already carries `required_sections`, `quality_checks`, and friends. The phantom `dependencies.yaml` reference survives only in two prose locations: `workflows/actions/frame.md` (validation-gate step) and `workflows/activities/01-frame/README.md` (artifact-directory description). No consumer script actually opens a `dependencies.yaml` file: `scripts/helix_align_check.py` parses the spec stack from markdown (PRD, FEAT, US, ADR) and does not touch `dependencies.yaml`; `workflows/actions/reconcile-alignment.md` references "dependencies" only as a conceptual notion. | +| Requirements | The catalog must have a single source of truth for artifact-type dependencies. The rubric must stop asking for a file the catalog does not produce. Any consumer that currently mentions `dependencies.yaml` must be repointed at `meta.yml` so the rubric and the workflow agree. | + +## Decision + +We ratify **`meta.yml.relationships`** as the canonical encoding of artifact-type +dependencies, and we drop `dependencies.yaml` from the artifact-type rubric. +There is no second file. There is no generated projection. + +- Type-level dependencies (`depends_on`, `informs`, peer relations) live in + `meta.yml` under `relationships`. The structured shape is already defined in + `workflows/artifact-schema.md` under "Recommended fields". +- Type-level validation (`required_sections`, `quality_checks`, + `pattern_checks`, `automated_checks`) lives in `meta.yml` under `validation`. +- Instance-level dependencies (one artifact instance depending on another) + continue to live in instance frontmatter under `ddx.depends_on`, exactly as + `workflows/artifact-schema.md` already specifies. ADR-004 does not change + that. + +**Key Points**: one source of truth (`meta.yml`) | no `dependencies.yaml` +generation step | consumer prose realigned to `meta.yml` + +## Consumer-side compatibility + +We choose **option (a): update consumers to read `meta.yml` directly** over +option (b): generate a transient `dependencies.yaml` projection at build/check +time. Rationale: + +- `scripts/helix_align_check.py` does not currently read `dependencies.yaml` at + all. It parses markdown (`prd.md`, `FEAT-*.md`, `US-*.md`, `ADR-*.md`) and + scans the code tree for `@covers` citations. There is no flat dependencies + projection to preserve. Option (b) would invent a consumer that does not + exist. +- `workflows/actions/reconcile-alignment.md` likewise does not open a + `dependencies.yaml` file. Its references to "dependencies" are conceptual, + not file-bound. +- The only two real `dependencies.yaml` mentions are prose in + `workflows/actions/frame.md` (validation-gate step) and + `workflows/activities/01-frame/README.md` (artifact-directory description). + Both are documentation of *where validation rules live*, not code that opens + a file. Repointing the prose at `meta.yml.validation` and + `meta.yml.relationships` is a textual edit, no runtime impact. +- Option (b) would add a build/check step, a generated file, and a drift + surface (projection out of sync with `meta.yml`) for no consumer benefit. + That is the opposite of the audit's goal. + +If a future consumer ever needs a flat dependencies graph, it can either parse +`meta.yml` directly or compose a projection at read time. That decision belongs +to that consumer, not to the catalog. + +## Alternatives + +| Option | Pros | Cons | Evaluation | +|--------|------|------|------------| +| Generate `dependencies.yaml` per artifact type from `meta.yml` as a build step | Preserves the historical rubric shape | Adds a generated file, a drift surface, and a build/check step with no consumer that needs it | Rejected: no consumer reads the file; cost without benefit | +| Hand-author both `dependencies.yaml` and `meta.yml.relationships` | Matches the historical rubric literally | Two sources of truth; 48 missing files; encodes the same data twice; drift inevitable | Rejected: contradicts single-source goal | +| **Ratify `meta.yml.relationships` as canonical and drop `dependencies.yaml` from the rubric** | One source of truth; matches what the catalog actually ships; no generation step; rubric stops lying about expected files | Existing prose in `frame.md` and `01-frame/README.md` must be updated in the same change | **Selected: smallest sufficient alignment** | + +## Consequences + +| Type | Impact | +|------|--------| +| Positive | Catalog has one place to look for artifact-type dependencies and one place to look for validation rules: `meta.yml`. | +| Positive | The rubric stops asking for a file that does not exist; the 48/48 absence reported by the audit is no longer a violation. | +| Positive | Workflow prose (`workflows/actions/frame.md`, `workflows/activities/01-frame/README.md`) is repointed at the file that actually carries validation rules, removing a documentation-to-reality gap. | +| Positive | No generated projection means no projection-drift class of failure to defend against. | +| Negative | The rubric line in `workflows/artifact-schema.md` ("Recommended fields") and the prose in `workflows/actions/frame.md` / `workflows/activities/01-frame/README.md` must be updated in lockstep with this ADR; any future doc that re-introduces `dependencies.yaml` must be rejected as drift. | +| Neutral | Instance-level dependencies in `ddx.depends_on` frontmatter are unaffected; this ADR is about type-level rubric, not instance-level graph. | + +## Risks + +| Risk | Prob | Impact | Mitigation | +|------|------|--------|------------| +| A future workflow re-introduces `dependencies.yaml` by force of habit | M | L | Audit gate: a grep for `dependencies.yaml` outside ADR-004 itself should return zero hits after this change | +| A future consumer wants a flat dependencies projection and reinvents it ad hoc | L | M | Consumers may compose a projection from `meta.yml` at read time; no canonical file required | +| The audit's "48/48 absent" finding gets re-reported because the rubric is read out of date | L | L | Update the rubric prose in the same commit as this ADR so the next audit sees the canonical shape | + +## Validation + +| Success Metric | Review Trigger | +|----------------|----------------| +| Grep for `dependencies.yaml` outside ADR-004 returns zero hits in `workflows/` and `scripts/` | A workflow doc or script re-introduces `dependencies.yaml` | +| `workflows/actions/frame.md` validation-gate step references `meta.yml.validation` (not `dependencies.yaml`) | A reviewer cannot tell where validation rules live for an artifact type | +| `workflows/activities/01-frame/README.md` artifact-directory description does not list `dependencies.yaml` | A new artifact type ships a `dependencies.yaml` instead of `meta.yml` updates | +| `workflows/artifact-schema.md` "Recommended fields" no longer implies a separate dependencies file | A consumer ships code that opens `dependencies.yaml` | + +## References + +- [Audit plan 2026-05-30](../plan-2026-05-30-artifact-types-and-concerns-audit.md) +- [PRD](../../01-frame/prd.md) +- [Artifact schema](../../../../workflows/artifact-schema.md) +- ADR-002: HELIX Tracker Write Safety Model +- ADR-003: Autonomy Spectrum diff --git a/docs/helix/02-design/adr/ADR-005-concern-practices-activity-keyed.md b/docs/helix/02-design/adr/ADR-005-concern-practices-activity-keyed.md new file mode 100644 index 00000000..e2e128c8 --- /dev/null +++ b/docs/helix/02-design/adr/ADR-005-concern-practices-activity-keyed.md @@ -0,0 +1,80 @@ +--- +ddx: + id: ADR-005 + depends_on: + - helix.prd +--- +# ADR-005: Concern practices.md is HELIX-activity-keyed + +| Date | Status | Deciders | Related | Confidence | +|------|--------|----------|---------|------------| +| 2026-05-30 | Accepted | HELIX maintainers | concerns library, context-digest assembly | High | + +## Context + +| Aspect | Description | +|--------|-------------| +| Problem | Concern `practices.md` files use two incompatible organizing principles. Some are keyed by HELIX activity (Discover / Frame / Design / Test / Build / Deploy / Iterate), so per-activity context-digest assembly is a mechanical lookup. Others are keyed by topic (Implementation / Quality Gates / etc.), forcing the context assembler to guess which heading belongs to which activity. | +| Current State | Ten concerns are topic-keyed: `auth-local-sessions`, `caching-strategy`, `classic-layered`, `cqrs`, `domain-driven-design`, `enterprise-application-patterns`, `hexagonal-architecture`, `mcp-server`, `onion-architecture`, `sample-data`. The remaining concerns already organize practices by HELIX activity. | +| Requirements | Per-activity context-digest assembly must be a deterministic lookup, not a heuristic. The concerns library must use one consistent organizing principle so that any consumer — context assembler, alignment review, methodology docs — extracts the practices for a given activity by reading the corresponding heading. | + +## Decision + +Every concern's `practices.md` organizes its per-activity content under HELIX +activity headings: **Discover**, **Frame**, **Design**, **Test**, **Build**, +**Deploy**, **Iterate**. Topic-keyed or quality-gates-keyed organization is no +longer acceptable for `practices.md`. + +Activity-keying makes per-activity context-digest assembly a mechanical lookup: +the assembler reads the heading whose name matches the current activity and +pulls the bullets underneath. No guessing, no mapping table, no per-concern +schema. + +The ten topic-keyed concerns named above convert to activity-keying in Phase 3 +of the artifact-types-and-concerns audit. New concerns ship activity-keyed from +day one. + +**Key Points**: HELIX activity headings are the only acceptable top-level +structure in `practices.md` | mechanical per-activity lookup | ten existing +topic-keyed concerns convert in Phase 3 + +## Alternatives + +| Option | Pros | Cons | Evaluation | +|--------|------|------|------------| +| Keep both styles (activity-keyed and topic-keyed) | No conversion work | Context-digest assembly must implement per-concern mapping logic and stay in sync as concerns evolve | Rejected: pushes guesswork into every consumer | +| Topic-keyed everywhere (Implementation / Quality Gates / ...) | Matches a few existing concerns | Forces the assembler to guess which topic applies to which activity; topics differ by concern | Rejected: not mechanically extractable | +| Add a frontmatter mapping table from topic → activity in each `practices.md` | Preserves topic prose | Introduces a per-concern schema to maintain; the heading and the mapping can drift | Rejected: heading is already the natural index | +| **Activity-keyed everywhere; convert the ten topic-keyed concerns in Phase 3** | One consistent structure; mechanical lookup; matches the majority of existing concerns | Requires per-concern restructuring for ten concerns | **Selected: smallest sufficient unification** | + +## Consequences + +| Type | Impact | +|------|--------| +| Positive | Per-activity context-digest assembly becomes a mechanical heading lookup across every concern. | +| Positive | Downstream consumers (alignment review, methodology docs, the website mirror) get a uniform structure to render and reason about. | +| Positive | New concerns have one obvious shape to follow; no per-concern bikeshed about structure. | +| Negative | The ten topic-keyed concerns require restructuring; some topic prose (e.g. cross-cutting Quality Gates) has to be reseated under the activity it applies to. | +| Neutral | The bullet content of each concern is preserved; only the organizing headings change. | + +## Risks + +| Risk | Prob | Impact | Mitigation | +|------|------|--------|------------| +| Conversion silently drops practice content during restructuring | M | M | Phase 3 conversion reviews the diff per concern; bullets must land under exactly one activity heading | +| Cross-cutting content (e.g. Quality Gates) has no natural single activity home | M | M | Allow the same bullet to appear under more than one activity heading when it genuinely applies across activities; prefer the closest activity otherwise | +| New concerns drift back to topic-keying | L | M | Lint or review check rejects `practices.md` whose top-level headings are not from the HELIX activity vocabulary | + +## Validation + +| Success Metric | Review Trigger | +|----------------|----------------| +| Every `practices.md` uses HELIX activity headings as its top-level structure | A `practices.md` lands or merges with topic-keyed headings | +| The ten named concerns are converted in Phase 3 with no lost content | Phase 3 closes with any of the ten still topic-keyed | +| Context-digest assembly extracts per-activity practices by heading lookup, with no per-concern mapping | The assembler grows per-concern conditional logic | + +## References + +- [PRD](../../01-frame/prd.md) +- Phase 1A plan: `docs/helix/02-design/plan-2026-05-30-artifact-types-and-concerns-audit.md` +- Concerns library: `workflows/concerns/` diff --git a/docs/helix/02-design/adr/ADR-006-concern-boundary-lives-once.md b/docs/helix/02-design/adr/ADR-006-concern-boundary-lives-once.md new file mode 100644 index 00000000..94495d5d --- /dev/null +++ b/docs/helix/02-design/adr/ADR-006-concern-boundary-lives-once.md @@ -0,0 +1,111 @@ +--- +ddx: + id: ADR-006 + depends_on: + - helix.prd +--- +# ADR-006: Concern boundary lives once, in concern.md + +| Date | Status | Deciders | Related | Confidence | +|------|--------|----------|---------|------------| +| 2026-05-30 | Accepted | HELIX maintainers | plan-2026-05-30-artifact-types-and-concerns-audit, ADR-004 (dependencies encoding), ADR-005 (practices format) | High | + +## Context + +| Aspect | Description | +|--------|-------------| +| Problem | The current concern file shape mandates three sections that naturally restate the same boundary: `concern.md`'s `Boundary` section, `practices.md`'s `Boundary with neighbors` section, and the `Constraints` / `Drift Signals` / `Quality Gates` sections that re-encode the same MUSTs in three different formats. Restating the boundary in three places is the structural cause of the verbosity flagged across the catalog — not editorial style. | +| Current State | The 2026-05-30 audit rated 11 of 48 artifact types and 22 of 49 concerns `verbose`. The concerns trace uniformly to this restatement pattern: the audit found `authorization-model`, `multi-tenancy`, `resilience`, `enterprise-integration-patterns`, `relational-data-modeling`, `verification`, `usage-metering`, `ux-radix`, `twelve-factor`, `event-sourcing`, `deployment-topology` all carrying the same Boundary text in three places. Editorial trims partially regress without a shape fix because the contract still asks for three boundary sections. The audit also surfaced two adjacent restatement vectors — `meta.yml` carrying taxonomies and process metadata that duplicate prompt/template content, and prompts/templates each carrying their own checklist of the same rules — that an editorial-only pass would not address. | +| Requirements | The catalog contract must define section ownership so the boundary is stated once, the other sections reference-not-restate it, and a schema validator can flag restatement as drift instead of relying on editorial review. The contract must also fold in the editorial rules from the dropped concision phase so the same contract that eliminates structural restatement also eliminates the adjacent vectors. | + +## Decision + +Every concern's boundary lives canonically in `concern.md`'s `Boundary` +section, and **only** there. The audit-named `Boundary with neighbors` +section in `practices.md` is removed; the `Constraints`, `Drift Signals`, +and `Quality Gates` sections that remain (in either file, per the +practices-format decision in ADR-005) **reference** the canonical boundary +rather than restating it. + +Restatement of boundary prose outside `concern.md`'s `Boundary` section is +now a **drift signal** the catalog schema validator (introduced in Phase 1 +Track B) can detect and flag. + +We also fold the editorial rules from the dropped concision phase into this +ADR so the contract that fixes the structural restatement also fixes the +adjacent vectors: + +1. **`meta.yml` is for machine validation + relationships only.** It does + not carry taxonomies, process metadata, decision frameworks, success + indicators, engagement levels, or other content that duplicates prompt + or template prose. Anything in `meta.yml` that is not consumed by a + validator or relationship resolver is drift. +2. **Prompt Quality Checklist and template Review Checklist are not both + present.** The catalog picks one home per artifact type; the other is + removed. (Default home: template Review Checklist for human-authored + artifacts; prompt Quality Checklist for generator-authored artifacts. + Per-artifact choice is recorded in `meta.yml`.) +3. **Tutorial-grade code blocks move to resource docs.** Full Dockerfiles, + complete Hugo configs, large CSS blocks, multi-step tmux scripts, and + similar long-form examples do not live in `concern.md` or + `practices.md`. They live under `docs/resources/` and the concern + references them by path. + +Each of these is enforceable by the same validator that flags boundary +restatement: presence of disallowed `meta.yml` keys, presence of both +checklists, or code blocks exceeding a size threshold become drift +signals. + +**Key Points**: boundary stated once in `concern.md` | other sections +reference-not-restate | restatement is a validator-flagged drift signal | +`meta.yml` is machine-only | one checklist per artifact | tutorial code +lives in resource docs + +## Alternatives + +| Option | Pros | Cons | Evaluation | +|--------|------|------|------------| +| Keep three boundary sections; rely on editorial review to keep them in sync | No contract change; no validator work | The 2026-05-30 audit shows editorial review does not keep them in sync — 22 concerns drifted into restatement; partial trims regress because the contract still asks for three sections | Rejected: this is the status quo the audit flagged | +| Put the boundary in `practices.md`'s `Boundary with neighbors` instead and have `concern.md`'s `Boundary` reference it | Keeps the neighbor-comparison framing | `concern.md` is the entry document a reader opens first; making it reference forward is worse ergonomics; the `practices.md` boundary section is the one that drifted, not `concern.md`'s | Rejected: wrong canonical home | +| Remove the boundary concept entirely; rely on Constraints / Drift Signals / Quality Gates | One fewer section type | The boundary is the load-bearing scope statement; the MUSTs are derived from it. Removing the boundary makes the MUSTs unmoored | Rejected: loses the source of truth | +| **Boundary lives once in `concern.md`; everything else references; validator flags restatement; fold editorial rules into the same contract** | Eliminates structural cause of restatement; mechanizable; one contract covers all three vectors | Requires validator work and a mechanical edit phase to remove the restated prose | **Selected: smallest sufficient structural fix** | + +## Consequences + +| Type | Impact | +|------|--------| +| Positive | The Phase 3 mechanical edits become a deterministic translation — remove the `Boundary with neighbors` section and rewrite Constraints / Drift Signals / Quality Gates as references — rather than a judgment call. | +| Positive | Future concerns can pass the catalog validator only if the boundary is stated once; the verbose-by-restatement pattern cannot reintroduce silently. | +| Positive | The validator that enforces single-boundary also enforces the `meta.yml` / dual-checklist / tutorial-code rules, so the dropped concision phase's intent is preserved without a separate editorial pass. | +| Positive | The catalog contract becomes one coherent policy — file shape, machine metadata, prose ownership, and resource separation are all enforced by the same validator. | +| Negative | Every existing concern needs the Phase 3 mechanical edit (remove `Boundary with neighbors`, rewrite three sections to reference the canonical boundary). | +| Negative | Several `meta.yml` files lose substantial content (the taxonomies and process metadata that duplicate prompt/template); this is a one-time migration. | +| Negative | The validator must define what "restatement" means precisely enough to catch real drift without false positives on legitimate cross-reference prose. | +| Neutral | The `practices.md` file remains; its body (Design / Implementation / MUST / SHOULD / Quality Gates, or the activity-keyed form per ADR-005) keeps its non-boundary content. | + +## Risks + +| Risk | Prob | Impact | Mitigation | +|------|------|--------|------------| +| Validator definition of "restatement" produces false positives on legitimate cross-references | M | M | Start with structural checks (presence of a `Boundary with neighbors` H2; `meta.yml` keys outside the allowlist; both checklists present) before attempting prose-similarity detection | +| Phase 3 edits remove prose readers actually relied on for the neighbor comparison | L | M | The canonical `Boundary` section can grow to subsume any load-bearing neighbor language; the rule is "stated once," not "stated minimally" | +| The `meta.yml` allowlist is too narrow and breaks existing tooling | M | M | Inventory consumers of `meta.yml` keys before defining the allowlist; the allowlist is the union of consumed keys | +| Tutorial code moved to `docs/resources/` loses context that made it useful in-line | L | L | Concerns reference the resource doc by path with a one-line summary of what the resource demonstrates | +| Choosing prompt vs template checklist per artifact reopens the orthogonality discussion the catalog is trying to close | L | M | Record the default rule (template checklist for human-authored, prompt for generator-authored) and require an explicit override in `meta.yml` only when deviating | + +## Validation + +| Success Metric | Review Trigger | +|----------------|----------------| +| The catalog schema validator flags any concern with a `Boundary with neighbors` H2 outside `concern.md`'s `Boundary` section | A new concern lands with restated boundary prose without the validator failing | +| Every `meta.yml` in the catalog contains only keys on the consumer-derived allowlist | A `meta.yml` key is added that no validator or relationship resolver reads | +| No artifact type ships both a prompt Quality Checklist and a template Review Checklist | A PR introduces or restores the duplicate checklist | +| Concern files contain no tutorial-grade code blocks (size threshold honored); tutorial content lives under `docs/resources/` and is referenced by path | A concern grows a multi-screen code example inline | +| Phase 3's mechanical edit pass removes the `Boundary with neighbors` section from every concern and rewrites Constraints / Drift Signals / Quality Gates as references, without regressing the canonical `Boundary` content | The audit re-run still flags the restatement pattern after Phase 3 lands | + +## References + +- [Plan: artifact-types-and-concerns audit (2026-05-30)](../plan-2026-05-30-artifact-types-and-concerns-audit.md) +- ADR-004: Dependencies encoding (Phase 1 catalog contract) +- ADR-005: Practices format (Phase 1 catalog contract) +- [PRD](/home/erik/Projects/helix/docs/helix/01-frame/prd.md) diff --git a/docs/helix/02-design/adr/ADR-007-empty-adr-references-headers-dropped.md b/docs/helix/02-design/adr/ADR-007-empty-adr-references-headers-dropped.md new file mode 100644 index 00000000..2c12f8a4 --- /dev/null +++ b/docs/helix/02-design/adr/ADR-007-empty-adr-references-headers-dropped.md @@ -0,0 +1,70 @@ +--- +ddx: + id: ADR-007 + depends_on: + - helix.prd +--- +# ADR-007: Empty ADR References Headers Are Dropped; Required Only When a Concern Cites a Specific ADR + +| Date | Status | Deciders | Related | Confidence | +|------|--------|----------|---------|------------| +| 2026-05-30 | Accepted | HELIX maintainers | concerns library, schema validator | High | + +## Context + +| Aspect | Description | +|--------|-------------| +| Problem | Many concerns ship an `ADR References` section header with no entries beneath it. The empty header creates a false promise of governing decisions that do not exist, and pads the artifact with structure that carries no information. | +| Current State | 14 concerns currently ship an empty `ADR References` header: `a11y-wcag-aa`, `demo-asciinema`, `demo-playwright`, `design-patterns-gof`, `e2e-kind`, `e2e-playwright`, `go-std`, `hugo-hextra`, `i18n-icu`, `k8s-kind`, `o11y-otel`, `python-uv`, `rust-cargo`, `scala-sbt`. The schema validator does not require the section to be present. | +| Requirements | The concerns library must not advertise references that do not exist. The section should appear only when it carries real ADR citations, and should be omitted when it would otherwise be empty. The schema validator must continue to treat the section as optional. | + +## Decision + +An `ADR References` section header in a concern is included **only when the +concern actually cites one or more ADRs**. Empty headers are removed by Phase 3 +mechanical edits across the 14 concerns enumerated above. The schema validator +does not require this section; presence is driven by content, not by template +shape. + +Concerns that genuinely govern decisions via ADRs keep the section and list the +ADRs they cite. Concerns with no real ADR citations omit the header entirely. + +**Key Points**: header presence follows content | empty headers removed in +Phase 3 | schema validator keeps the section optional + +## Alternatives + +| Option | Pros | Cons | Evaluation | +|--------|------|------|------------| +| Keep empty `ADR References` headers as a template slot | Uniform structure across concerns | False promise of citations; padding without information; invites cargo-cult copy of the empty section into new concerns | Rejected: structure without content is noise | +| Require every concern to cite at least one ADR | Forces explicit governance | Most concerns are not governed by an ADR; would manufacture spurious references | Rejected: not all concerns need ADR backing | +| **Drop empty headers; require the section only when ADRs are cited** | Removes false promise; keeps real citations visible; matches existing validator behavior | Requires a one-time mechanical edit pass across 14 concerns | **Selected: smallest change that aligns shape with content** | + +## Consequences + +| Type | Impact | +|------|--------| +| Positive | Concerns no longer advertise references that do not exist. | +| Positive | Real ADR citations remain visible and unambiguous in the concerns that have them. | +| Positive | New concerns inherit a "populate or omit" rule that prevents the empty header from reappearing. | +| Negative | Phase 3 must touch 14 concern files to remove the empty headers. | +| Neutral | The schema validator is unchanged; the section stays optional. | + +## Risks + +| Risk | Prob | Impact | Mitigation | +|------|------|--------|------------| +| Future concerns reintroduce the empty header by copy-paste | M | L | Phase 3 edits remove existing instances; reviewers reject empty headers on new concerns | +| A real ADR citation is accidentally removed alongside the empty headers | L | M | Phase 3 mechanical edits target only the enumerated 14 concerns where the section is empty | + +## Validation + +| Success Metric | Review Trigger | +|----------------|----------------| +| None of the 14 enumerated concerns ships an empty `ADR References` header after Phase 3 | An empty `ADR References` header appears in any concern | +| Concerns that cite ADRs retain the section with the citations intact | A real ADR citation is dropped by the Phase 3 edits | + +## References + +- [PRD](../../01-frame/prd.md) +- [Plan: 2026-05-30 Artifact Types and Concerns Audit](../plan-2026-05-30-artifact-types-and-concerns-audit.md) diff --git a/docs/helix/02-design/adr/ADR-008-data-prd-is-prd-kind-variant.md b/docs/helix/02-design/adr/ADR-008-data-prd-is-prd-kind-variant.md new file mode 100644 index 00000000..6a0e75b5 --- /dev/null +++ b/docs/helix/02-design/adr/ADR-008-data-prd-is-prd-kind-variant.md @@ -0,0 +1,76 @@ +--- +ddx: + id: ADR-008 + depends_on: + - helix.prd +--- +# ADR-008: data-PRD Is a Kind-Switch Variant of PRD, Not a Sibling Artifact + +| Date | Status | Deciders | Related | Confidence | +|------|--------|----------|---------|------------| +| 2026-05-30 | Accepted | HELIX maintainers | artifact-types catalog, prd, data-prd | High | + +## Context + +| Aspect | Description | +|--------|-------------| +| Problem | The `data-prd` artifact type is a near-duplicate of `prd` with data-product framing. Its `required_sections` drift from the `prd` template, the boundary between the two artifact types is not load-bearing, and the catalog carries an orthogonality defect: two artifact-type directories that differ only in framing, not in role. | +| Current State | `workflows/activities/01-frame/artifacts/prd/` and `workflows/activities/01-frame/artifacts/data-prd/` ship as sibling artifact-type directories. Both serve the framing role of "product requirements document"; `data-prd` only adjusts vocabulary toward data products. The two trees have diverged in required sections without a substantive reason. | +| Requirements | The catalog must represent one PRD role with explicit variant framing rather than two parallel artifact types. The variant must remain discoverable for data-product framing without forcing every PRD to carry data-product vocabulary. Downstream consumers that reference `data-prd` must continue to resolve. | + +## Decision + +`data-prd` collapses into `prd` as a `kind: data` variant. The PRD artifact +gains a `kind` field (default `product`); when set to `data`, the prompt and +template guidance shift to data-product framing under conditional headings. +The current `data-prd` artifact-type directory is deprecated and will be +removed in Phase 3. + +The PRD artifact owns one role — framing product (or data-product) +requirements — and exposes the variant through a kind switch on its meta. The +shape of the artifact is unified; the framing is parameterized. + +**Key Points**: one PRD artifact with a `kind` switch | `kind: product` +default, `kind: data` for data products | `data-prd` directory removed in +Phase 3 with alias for downstream consumers + +## Alternatives + +| Option | Pros | Cons | Evaluation | +|--------|------|------|------------| +| Keep `data-prd` as a sibling artifact type | No migration work; existing references unchanged | Perpetuates orthogonality defect; required-section drift continues; catalog advertises a boundary that is not load-bearing | Rejected: the boundary does not carry weight | +| Merge `data-prd` into `prd` with no variant marker | Simplest collapse | Loses data-product framing entirely; existing data-PRD consumers lose guidance | Rejected: framing variant has real value | +| **Collapse `data-prd` into `prd` as a `kind: data` variant** | Unifies the role; preserves the framing as a parameterized variant; eliminates the duplicate directory; lets the template carry conditional guidance | Phase 3 must edit `prd` meta/prompt/template and remove the `data-prd` directory; downstream references to `data-prd` must be redirected or aliased | **Selected: smallest change that resolves the defect while preserving the framing variant** | + +## Consequences + +| Type | Impact | +|------|--------| +| Positive | The artifact-type catalog represents one PRD role instead of two near-duplicates. | +| Positive | Required-section drift between `prd` and `data-prd` ends; one template governs both framings. | +| Positive | Data-product framing remains discoverable through an explicit `kind: data` switch rather than a parallel directory. | +| Positive | New PRD-shaped variants (if any) can be added as additional `kind` values without forking the directory. | +| Negative | Phase 3 must extend `prd` meta/prompt/template, delete the `data-prd` directory, and update or alias every reference under `docs/helix/` and `workflows/`. | +| Neutral | Existing PRDs default to `kind: product` and require no content change. | + +## Risks + +| Risk | Prob | Impact | Mitigation | +|------|------|--------|------------| +| Downstream consumers still resolve `data-prd` by path | M | M | Add an alias in any downstream consumer that resolves artifact-type paths; update references in the same Phase 3 pass | +| Conditional kind-specific guidance in the template becomes hard to read | M | M | Keep kind-specific guidance under clearly labeled conditional headings; avoid interleaving product and data framing line-by-line | +| New PRD variants accumulate via the kind switch without governance | L | M | Document the kind enum in `prd` meta with the supported values; new values require an ADR | + +## Validation + +| Success Metric | Review Trigger | +|----------------|----------------| +| `workflows/activities/01-frame/artifacts/data-prd/` is removed after Phase 3 | A `data-prd` artifact-type directory remains after Phase 3 | +| `prd` meta declares a `kind` enum with `product` (default) and `data` values | The `prd` artifact ships without a `kind` field after Phase 3 | +| `prd` prompt and template carry kind-specific guidance under conditional headings | Data-product framing is lost from the unified PRD artifact | +| All references to `data-prd` under `docs/helix/` and `workflows/` resolve to `prd` with the kind switch | A reference to `data-prd` resolves to a missing directory after Phase 3 | + +## References + +- [PRD](../../01-frame/prd.md) +- [Plan: 2026-05-30 Artifact Types and Concerns Audit](../plan-2026-05-30-artifact-types-and-concerns-audit.md) diff --git a/docs/helix/02-design/adr/ADR-009-acceptance-criteria-ownership.md b/docs/helix/02-design/adr/ADR-009-acceptance-criteria-ownership.md new file mode 100644 index 00000000..a2cc1d07 --- /dev/null +++ b/docs/helix/02-design/adr/ADR-009-acceptance-criteria-ownership.md @@ -0,0 +1,75 @@ +--- +ddx: + id: ADR-009 + depends_on: + - helix.prd +--- +# ADR-009: Acceptance Criteria Live in User-Stories; Feature-Specification Owns Functional Areas and Decomposition + +| Date | Status | Deciders | Related | Confidence | +|------|--------|----------|---------|------------| +| 2026-05-30 | Accepted | HELIX maintainers | user-stories, feature-specification | High | + +## Context + +| Aspect | Description | +|--------|-------------| +| Problem | The rule "AC-IDs live in user-stories" is restated five times across the `user-stories` artifact (meta `quality_checks`, meta `automated_checks`, prompt section, prompt blocking checklist, template) and partly mirrored on `feature-specification`. The boundary "FEAT owns FR-n; stories own AC-IDs" is asserted in multiple places but is not load-bearing in any single one, which makes it easy for a future edit to drop the rule on one side without noticing. | +| Current State | `feature-specification` includes acceptance-criteria guidance in its prompt and template (covering happy paths, errors, edge cases with observable outcomes) alongside its FR-n and functional-area material. `user-stories` carries the canonical AC-ID rule but duplicates it across five surfaces. The split of ownership between the two artifacts is implicit rather than declared. | +| Requirements | The HELIX frame phase must have exactly one artifact that owns acceptance-criteria definition and AC-IDs. The boundary must be stated once, load-bearing, and survive future edits. Feature-specification must remain the home of functional areas and the decomposition test (which stories the feature decomposes into) without restating AC material. | + +## Decision + +Acceptance criteria and AC-IDs live exclusively in `user-stories`. +`feature-specification` declares FR-IDs, functional areas, and the +decomposition test (which user stories the feature decomposes into) but does +**not** define or restate acceptance criteria. + +Phase 3 will remove AC-ID and acceptance-criteria restatement from +`feature-specification`'s prompt and template. `user-stories` keeps the single +canonical AC-ID rule, with the meta-quality-check duplication trimmed so the +rule lives in one place rather than five. + +**Key Points**: AC-IDs owned by `user-stories` | `feature-specification` owns +FR-IDs, functional areas, and decomposition | rule stated once, not echoed +across surfaces + +## Alternatives + +| Option | Pros | Cons | Evaluation | +|--------|------|------|------------| +| Keep AC material on both artifacts and rely on cross-references | Uniform appearance; each artifact looks self-contained | Five-way restatement on `user-stories` plus partial mirror on `feature-specification`; no single load-bearing statement; future edits silently drift one side | Rejected: duplication without authority is the current failure mode | +| Move AC ownership to `feature-specification` instead | Feature is the higher-level container; AC near FRs reads naturally | Stories become weaker as test contracts; AC-IDs lose their natural scope (US-``-AC``); breaks the decomposition direction (feature -> stories -> AC) | Rejected: AC-ID identity is story-scoped, not feature-scoped | +| **AC-IDs on `user-stories` only; `feature-specification` owns FR-IDs and decomposition** | Single owner per concept; load-bearing rule; preserves the FR -> story -> AC chain; matches how AC-IDs are already scoped | Requires Phase 3 mechanical edits to `feature-specification` prompt/template and trimming `user-stories` meta duplication | **Selected: smallest change that makes the boundary load-bearing** | + +## Consequences + +| Type | Impact | +|------|--------| +| Positive | The "AC-IDs live in user-stories" rule is stated once and is load-bearing rather than echoed five ways. | +| Positive | `feature-specification` focuses on FR-IDs, functional areas, and the decomposition test, which is its actual job. | +| Positive | The FR -> story -> AC decomposition direction is visible in the artifact shapes themselves. | +| Positive | Future edits cannot silently drift the boundary; there is one place to change. | +| Negative | Phase 3 must touch `feature-specification`'s prompt and template to remove acceptance-criteria material, and trim `user-stories` meta to collapse the five-way restatement. | +| Neutral | The FR-n / AC-ID identifier schemes themselves are unchanged. | + +## Risks + +| Risk | Prob | Impact | Mitigation | +|------|------|--------|------------| +| `feature-specification` authors continue to write AC inline by habit | M | M | Phase 3 prompt edits redirect AC material to `user-stories`; reviewers reject AC in feature-specification | +| Removing AC guidance from `feature-specification` weakens feature-level testability framing | L | M | Decomposition test stays on feature-specification and points reviewers at the story-level AC for testability | +| `user-stories` meta trimming drops a check that was actually catching defects | L | M | Phase 3 keeps the single canonical AC-ID rule; only redundant restatements are removed | + +## Validation + +| Success Metric | Review Trigger | +|----------------|----------------| +| `feature-specification` prompt and template contain no AC-ID or acceptance-criteria definitions after Phase 3 | Acceptance-criteria material reappears in `feature-specification` | +| `user-stories` states the AC-ID rule in exactly one canonical place | The AC-ID rule is restated across multiple surfaces of `user-stories` again | +| `feature-specification` carries the decomposition test (which stories the feature decomposes into) and FR-IDs as its primary content | A feature is filed without the decomposition test or FR-IDs | + +## References + +- [PRD](../../01-frame/prd.md) +- [Plan: 2026-05-30 Artifact Types and Concerns Audit](../plan-2026-05-30-artifact-types-and-concerns-audit.md) diff --git a/docs/helix/02-design/adr/ADR-010-feature-registry-parking-lot-handoff.md b/docs/helix/02-design/adr/ADR-010-feature-registry-parking-lot-handoff.md new file mode 100644 index 00000000..bc29619c --- /dev/null +++ b/docs/helix/02-design/adr/ADR-010-feature-registry-parking-lot-handoff.md @@ -0,0 +1,73 @@ +# ADR-010: feature-registry and parking-lot stay separate with an explicit handoff + +| Date | Status | Deciders | Related | Confidence | +|------|--------|----------|---------|------------| +| 2026-05-30 | Accepted | HELIX maintainers | feature-registry, parking-lot, frame-phase catalog | High | + +## Context + +| Aspect | Description | +|--------|-------------| +| Problem | The artifact-type audit flagged `feature-registry` and `parking-lot` as similarly-shaped registries in the frame phase, raising the question of whether they should collapse into a single registry distinguished by a status enum. | +| Current State | `feature-registry` is the source of truth for accepted features (`FEAT-XXX`, owners, priorities, status progression from `draft` through `deployed`/`deprecated`/`cancelled`). `parking-lot` is a separate registry for deferred or future work, keyed by `Source` and `Revisit Trigger` and explicitly outside current scope. They share a `type: registry` classification and live in the same activity, but encode different lifecycle states and serve different operator decisions. | +| Requirements | The catalog must either (a) merge the two into one registry with a status discriminator, or (b) keep them separate with a documented handoff so the boundary the operator triages on stays visible and tracker-relevant. | + +## Decision + +`feature-registry` and `parking-lot` remain separate artifacts. + +`parking-lot` is the staging area for deferred or future work — items kept +visible but outside current scope until a recorded `Revisit Trigger` fires. +`feature-registry` is the source of truth for accepted features that are in +the active pipeline (`draft` → `specified` → `designed` → `in_test` → +`in_build` → `built` → `deployed`, or terminal `deprecated`/`cancelled`). + +Handoff between the two is explicit: a parking-lot entry promotes to +feature-registry when its revisit criteria are met. The promotion is a +recorded transition (assign next sequential `FEAT-XXX`, set initial status, +link back to the parking-lot source), not an implicit merge or status flip +inside a single registry. + +**Key Points**: deferred vs accepted is a real lifecycle boundary | promotion +is a recorded transition, not an implicit merge | the operator triages on the +boundary, so the boundary must be visible in the artifact shape + +## Alternatives + +| Option | Pros | Cons | Evaluation | +|--------|------|------|------------| +| Collapse into one registry with a `deferred` status value | One file, one schema, one prompt; removes "similar shape" duplication | Loses the triage boundary the operator uses; mixes accepted features (with owners, priorities, dependencies) with deferred items (with revisit triggers); forces every consumer to filter by status before reading | Rejected: collapses a real lifecycle distinction | +| Keep separate, leave handoff implicit (current state) | No new procedure to document | Promotion happens ad-hoc; no recorded link between parked source and registered feature; the audit's "same shape" concern persists | Rejected: the handoff is the thing that makes separation safe | +| **Keep separate, document an explicit promotion procedure** | Preserves the lifecycle boundary; makes the transition auditable; addresses the audit's concern without merging | Requires Phase 3 to document the promotion procedure once and cross-reference it from parking-lot | **Selected: smallest sufficient fix that keeps the boundary real** | + +## Consequences + +| Type | Impact | +|------|--------| +| Positive | The deferred/accepted boundary stays visible in the artifact shape, matching how operators triage. | +| Positive | Promotion becomes a recorded transition with a back-link, so traceability survives the handoff. | +| Positive | The audit's "same shape" concern is resolved by procedure, not by losing structure. | +| Negative | Phase 3 owes the promotion procedure write-up in `feature-registry`, with a cross-reference from `parking-lot`. | +| Neutral | Both artifacts keep their existing `type: registry` classification and their existing prompts/templates. | + +## Risks + +| Risk | Prob | Impact | Mitigation | +|------|------|--------|------------| +| Promotion procedure documented in one place but not cross-referenced from the other | M | M | Phase 3 acceptance requires both ends — feature-registry owns the procedure, parking-lot links to it | +| Operators bypass the procedure and recreate parked items in the registry without back-links | M | M | The promotion step records the parking-lot source as the first dependency/trace link on the new FEAT-XXX | +| Future audit re-raises the "same shape" question after the procedure rots | L | L | This ADR is the durable answer; future audits cite it instead of relitigating | + +## Validation + +| Success Metric | Review Trigger | +|----------------|----------------| +| feature-registry documents the promotion procedure exactly once | Procedure is duplicated in parking-lot rather than cross-referenced | +| parking-lot's prompt cross-references the promotion procedure | A parked item is promoted without a recorded back-link to its parking-lot source | +| Promoted features carry a recorded link back to the parking-lot entry that sourced them | Operator triage cannot distinguish "deferred" from "accepted" at a glance | + +## References + +- [Plan: artifact-types and concerns audit](/Users/erik/Projects/helix/docs/helix/02-design/plan-2026-05-30-artifact-types-and-concerns-audit.md) +- `workflows/activities/01-frame/artifacts/feature-registry/` +- `workflows/activities/01-frame/artifacts/parking-lot/` diff --git a/docs/helix/02-design/adr/ADR-011-ux-radix-owns-shadcn-prescription.md b/docs/helix/02-design/adr/ADR-011-ux-radix-owns-shadcn-prescription.md new file mode 100644 index 00000000..44f562c2 --- /dev/null +++ b/docs/helix/02-design/adr/ADR-011-ux-radix-owns-shadcn-prescription.md @@ -0,0 +1,77 @@ +# ADR-011: ux-radix owns Radix and shadcn prescriptions; react-nextjs references + +| Date | Status | Deciders | Related | Confidence | +|------|--------|----------|---------|------------| +| 2026-05-30 | Accepted | HELIX maintainers | ux-radix, react-nextjs, concerns | High | + +## Context + +| Aspect | Description | +|--------|-------------| +| Problem | Both the `react-nextjs` concern and the `ux-radix` concern prescribe Radix primitives and the shadcn component library. This double-prescription confuses the concern boundary: which concern owns component-library choice, and which one is being referenced? Audits surface the duplication as drift rather than as a deliberate cross-reference. | +| Current State | `react-nextjs` (framework slot) and `ux-radix` (interaction-primitives slot) each independently prescribe shadcn/Radix in their `concern.md` and `practices.md`. There is no single canonical owner, and downstream consumers cannot tell which concern's prescription is authoritative. | +| Requirements | Component-library prescription must live in exactly one concern, with other concerns referencing the owner rather than re-prescribing. The owner should be the concern whose subject matter is interaction primitives and component libraries. | + +## Decision + +The `ux-radix` concern owns the Radix primitives and shadcn component-library +prescription. `react-nextjs` references `ux-radix` for UI primitives and does +NOT prescribe shadcn directly. + +`ux-radix` is the interaction-primitives concern; UI and component-library +prescription naturally lives there. `react-nextjs` is the framework slot +occupant — it should describe the framework (Next.js conventions, routing, +rendering model, build/runtime), not the component library it happens to pair +with. Removing component-library prescription from `react-nextjs` keeps the +framework concern focused on framework-shaped decisions. + +This follows the "concern boundary lives once" rule established in ADR-006: +every prescription has exactly one canonical owner, and other concerns +reference rather than re-state. + +**Key Points**: ux-radix is the canonical owner of Radix + shadcn prescription +| react-nextjs references ux-radix instead of re-prescribing | framework +concerns describe frameworks, not component libraries + +## Alternatives + +| Option | Pros | Cons | Evaluation | +|--------|------|------|------------| +| Keep double-prescription in both concerns | No edits required | Violates ADR-006 single-owner rule; auditors cannot tell which prescription is canonical; drift between the two copies is inevitable | Rejected: known source of confusion | +| Move ownership to `react-nextjs` and have `ux-radix` reference it | Co-locates component-library choice with the framework that pairs with it | Conflates framework slot with interaction-primitives slot; forces every framework concern to re-prescribe its own component library; ux-radix loses its subject matter | Rejected: wrong slot for the prescription | +| **Move ownership to `ux-radix`; `react-nextjs` references it** | Matches concern subject matter (interaction primitives own component libraries); keeps framework concerns focused; preserves slot boundaries | Requires Phase 3 edits to `react-nextjs` to remove or convert its shadcn prescription | **Selected: aligns ownership with concern subject** | + +## Consequences + +| Type | Impact | +|------|--------| +| Positive | Single canonical owner for shadcn/Radix prescription; no duplication to drift between. | +| Positive | `react-nextjs` becomes a focused framework concern instead of a mixed framework + component-library concern. | +| Positive | Future framework concerns (e.g. other React meta-frameworks) can reference `ux-radix` without re-prescribing the component library. | +| Positive | Audit drift signal becomes meaningful: any future shadcn/Radix mention outside `ux-radix` is a real cross-reference, not an accidental duplicate. | +| Negative | Phase 3 must edit `react-nextjs` to remove the shadcn/Radix prescription and replace it with a cross-reference. | +| Negative | Downstream artifacts that cite `react-nextjs` for component-library choice must be updated to cite `ux-radix` instead. | +| Neutral | The pairing of Next.js with shadcn/Radix in practice is unchanged — only the ownership of the prescription moves. | + +## Risks + +| Risk | Prob | Impact | Mitigation | +|------|------|--------|------------| +| Phase 3 edits to `react-nextjs` leave dangling shadcn references | M | M | Audit `react-nextjs/concern.md` and `practices.md` for any remaining shadcn/Radix prescriptions after the edit | +| Other framework concerns silently re-prescribe shadcn | L | M | Drift checker scans non-`ux-radix` concerns for shadcn/Radix prescription language | +| Cross-reference from `react-nextjs` to `ux-radix` is too vague to be useful | M | L | Make the reference explicit: name `ux-radix` and the specific primitives concern, not a generic "see UI concerns" pointer | + +## Validation + +| Success Metric | Review Trigger | +|----------------|----------------| +| Only `ux-radix` prescribes shadcn/Radix; all other concerns reference it | A non-`ux-radix` concern re-prescribes shadcn or Radix primitives | +| `react-nextjs` describes Next.js framework concerns without component-library prescription | `react-nextjs` adds component-library or design-system prescription | +| Downstream artifacts that need component-library choice cite `ux-radix` | An artifact cites `react-nextjs` for shadcn/Radix choice | + +## References + +- ADR-006: Concern boundary lives once +- workflows/concerns/ux-radix/concern.md +- workflows/concerns/react-nextjs/concern.md +- docs/helix/02-design/plan-2026-05-30-artifact-types-and-concerns-audit.md diff --git a/docs/helix/02-design/adr/ADR-012-runbook-owns-incident-response.md b/docs/helix/02-design/adr/ADR-012-runbook-owns-incident-response.md new file mode 100644 index 00000000..410d428a --- /dev/null +++ b/docs/helix/02-design/adr/ADR-012-runbook-owns-incident-response.md @@ -0,0 +1,84 @@ +--- +ddx: + id: ADR-012 + depends_on: + - helix.prd +--- +# ADR-012: Runbook owns incident response procedures; monitoring-setup owns detection only + +| Date | Status | Deciders | Related | Confidence | +|------|--------|----------|---------|------------| +| 2026-05-30 | Accepted | HELIX maintainers | plan-2026-05-30-artifact-types-and-concerns-audit, monitoring-setup, runbook | High | + +## Context + +| Aspect | Description | +|--------|-------------| +| Problem | The 2026-05-30 artifact-types-and-concerns audit flagged an ownership collision between `monitoring-setup` and `runbook`: both currently define incident-response routing. `monitoring-setup`'s template carries an `Incident Response` section (escalation paths, response procedures) that duplicates and competes with `runbook`'s `Common Incident Procedures` and escalation content. With two artifact types claiming the same surface, operators must reconcile two sources of truth at the moment they can least afford to. | +| Current State | `monitoring-setup/template.md` includes an `## Incident Response` section. `runbook/template.md` includes `## Common Incident Procedures` (with per-incident sections and a security/data-safety incident path) and explicit escalation routing. The two artifact types overlap on response procedures and escalation rather than partitioning detection from response. | +| Requirements | The catalog must assign incident-response ownership to exactly one artifact type. The other type must stop defining that surface so operators have a single canonical procedure document during an incident. | + +## Decision + +`monitoring-setup` owns **detection only**: SLI/SLO definitions, alert +routing inputs, threshold tuning, and the observability surface that +produces signals. + +`runbook` owns **incident response**: the procedures operators run when +those signals fire — recovery steps, common incident procedures, +escalation paths, and incident commander routing. + +In Phase 3, the `Incident Response` section is removed from the +`monitoring-setup` template. Any content from that section not already +covered moves to `runbook`. After Phase 3, the catalog validator can +treat presence of an `Incident Response` (or equivalent response-procedure) +H2 in `monitoring-setup` as a drift signal. + +**Key Points**: monitoring-setup = detection surface | runbook = operator +procedure | Phase 3 removes Incident Response from monitoring-setup +template | content not already in runbook moves there + +## Alternatives + +| Option | Pros | Cons | Evaluation | +|--------|------|------|------------| +| Keep both artifact types defining incident response; rely on authors to keep them in sync | No catalog change | The audit found they are already out of sync; two sources of truth during an incident is the worst possible time for ambiguity | Rejected: status quo the audit flagged | +| Move detection ownership into `runbook` and remove `monitoring-setup` | Single artifact type for the whole operational surface | Detection (SLI/SLO definitions, alert thresholds) and response (procedures, escalation) are authored by different people at different cadences; collapsing them loses that separation | Rejected: conflates two genuinely distinct authoring surfaces | +| **`monitoring-setup` owns detection; `runbook` owns response; Phase 3 removes the Incident Response section from monitoring-setup** | Clean partition aligned to the actual purpose of each artifact type; single source of truth for procedures; enforceable by validator | Requires a content migration for any monitoring-setup content not already in runbook | **Selected: smallest sufficient ownership fix** | + +## Consequences + +| Type | Impact | +|------|--------| +| Positive | Operators have a single canonical document (`runbook`) for incident procedures and escalation. | +| Positive | `monitoring-setup` becomes focused on its actual purpose — the observability surface — without bleeding into operator procedure. | +| Positive | The catalog validator can flag any future reintroduction of response procedures into `monitoring-setup` as drift. | +| Positive | Authoring cadences are no longer entangled: detection thresholds can evolve without re-touching response procedures, and vice versa. | +| Negative | Phase 3 must perform a content migration from `monitoring-setup`'s Incident Response section to `runbook` for anything not already covered. | +| Negative | Existing `monitoring-setup` artifacts authored under the previous contract need a one-time edit to remove the Incident Response section. | +| Neutral | The two artifact types continue to live side-by-side in the `05-deploy` activity; only their boundaries change. | + +## Risks + +| Risk | Prob | Impact | Mitigation | +|------|------|--------|------------| +| Content in `monitoring-setup`'s Incident Response section is lost rather than migrated | L | H | Phase 3 migration explicitly diffs the removed content against the runbook template before deleting; anything not already covered is added to runbook | +| Authors continue to add response procedures to `monitoring-setup` after Phase 3 | M | M | Catalog validator flags `Incident Response` (or equivalent response-procedure) H2 in `monitoring-setup` as drift | +| The detection/response partition is unclear at the boundary (e.g. alert routing) | M | M | Detection ends at "signal produced and routed to a destination"; response begins at "operator receives signal and acts." Alert routing destinations (PagerDuty, channel) are detection; what the recipient does is response | +| Existing references from outside the catalog point to the removed section | L | L | Phase 3 ships a redirect note in commit message and updates in-tree references in the same change | + +## Validation + +| Success Metric | Review Trigger | +|----------------|----------------| +| `monitoring-setup/template.md` contains no Incident Response section after Phase 3 | A PR reintroduces an Incident Response H2 (or equivalent response-procedure section) into `monitoring-setup` | +| All response-procedure content from the removed section is present in `runbook` (either pre-existing or migrated) | Phase 3 lands without a diff confirming migration coverage | +| The catalog validator flags `monitoring-setup` artifacts that carry response-procedure sections | A monitoring-setup artifact ships with response-procedure content and validation passes | +| Operators consulting `runbook` during an incident find escalation and procedure content without needing to cross-reference `monitoring-setup` | An incident retrospective surfaces split-source-of-truth as a contributing factor | + +## References + +- [Plan: artifact-types-and-concerns audit (2026-05-30)](../plan-2026-05-30-artifact-types-and-concerns-audit.md) +- [PRD](/home/erik/Projects/helix/docs/helix/01-frame/prd.md) +- `workflows/activities/05-deploy/artifacts/monitoring-setup/template.md` +- `workflows/activities/05-deploy/artifacts/runbook/template.md` diff --git a/docs/helix/02-design/adr/ADR-013-cross-phase-informs-edges.md b/docs/helix/02-design/adr/ADR-013-cross-phase-informs-edges.md new file mode 100644 index 00000000..ff046824 --- /dev/null +++ b/docs/helix/02-design/adr/ADR-013-cross-phase-informs-edges.md @@ -0,0 +1,78 @@ +--- +ddx: + id: ADR-013 + depends_on: + - helix.prd +--- +# ADR-013: Cross-phase informs edges are valid when declared on the downstream artifact + +| Date | Status | Deciders | Related | Confidence | +|------|--------|----------|---------|------------| +| 2026-05-30 | Accepted | HELIX maintainers | artifact dependencies, schema validator | High | + +## Context + +| Aspect | Description | +|--------|-------------| +| Problem | The audit flagged the Deploy artifact `monitoring-setup` declaring `informs: metrics-dashboard`, an Iterate artifact in a later phase. It is unclear whether HELIX permits an artifact to declare an `informs` edge that crosses a phase boundary into a downstream phase, and whether the back-edge must also be declared on the consumer. | +| Current State | `workflows/activities/05-deploy/artifacts/monitoring-setup/meta.yml` declares `informs: metrics-dashboard`. `workflows/activities/06-iterate/artifacts/metrics-dashboard/meta.yml` declares both `informed_by` and `informs` lists but does not currently encode every back-edge for every upstream `informs`. The schema validator does not enforce edge symmetry across artifacts. | +| Requirements | The methodology must not forbid real evidence-flow edges that cross phases (production telemetry feeding dashboards is a real flow). At the same time, cross-phase edges must be explicit on both ends so that walking from either side reveals the relationship. | + +## Decision + +An artifact may declare `informs` an artifact in a later phase. The cross-phase +edge is valid when: + +1. The source artifact's purpose produces evidence the target consumes (the + edge encodes a real evidence flow, not a speculative association), and +2. The edge is declared on both sides — `informs` on the source artifact and + `informed_by` on the target artifact — or in a single canonical place + chosen by the consumer when only one direction can be authoritative. + +The `monitoring-setup` → `metrics-dashboard` edge is the canonical example: +production telemetry configured in Deploy feeds the dashboards consumed in +Iterate. The methodology accepts this as a valid cross-phase informs edge. + +**Key Points**: cross-phase informs allowed | edge must be real evidence flow | +declared on both ends (or canonical single place) | symmetry check is a +follow-up validator concern, not required by this ADR + +## Alternatives + +| Option | Pros | Cons | Evaluation | +|--------|------|------|------------| +| Forbid cross-phase informs edges entirely; restrict `informs` to same-phase artifacts | Simpler validator; clear phase boundary | Erases real evidence flows (Deploy telemetry → Iterate dashboards); forces artifacts to invent intermediate hops or drop the edge | Rejected: phase boundary is not a methodology firewall against evidence flow | +| Allow cross-phase informs but only when declared on the upstream side | Single source of truth | Walking from the consumer no longer reveals the upstream; reviewers must scan all earlier phases to find inbound edges | Rejected: breaks bidirectional discoverability | +| **Allow cross-phase informs; require declaration on both ends (or a canonical single place); leave symmetry enforcement as a follow-up validator concern** | Preserves real evidence flows; keeps edges discoverable from both ends; does not block this ADR on validator work | Phase 3 walk must check both sides; some artifacts will need back-edges added | **Selected: accepts the real flow without overcommitting the validator** | + +## Consequences + +| Type | Impact | +|------|--------| +| Positive | Real cross-phase evidence flows (e.g., Deploy telemetry → Iterate dashboards) are recognized as first-class edges. | +| Positive | Both-end declaration keeps the dependency graph walkable from either direction. | +| Positive | Phase 3 has a clear rule: walk all artifacts for `informs` edges and ensure the back-edge is declared. | +| Negative | Phase 3 must check edge symmetry across the full artifact set and add missing back-edges where the upstream declares `informs` but the downstream does not declare `informed_by`. | +| Neutral | The schema validator is unchanged in this ADR. Symmetry enforcement is a follow-up Phase 1B validator concern. | + +## Risks + +| Risk | Prob | Impact | Mitigation | +|------|------|--------|------------| +| Authors declare speculative cross-phase edges that are not real evidence flows | M | M | Phase 3 walk and reviewers reject edges that do not match the source artifact's stated purpose | +| Back-edges drift out of sync as artifacts evolve | M | M | Follow-up Phase 1B validator extension checks edge symmetry; until then, the Phase 3 walk establishes the baseline | +| Confusion about whether to declare on one end or both | L | L | This ADR states the rule: both ends, or a single canonical place chosen by the consumer when only one direction can be authoritative | + +## Validation + +| Success Metric | Review Trigger | +|----------------|----------------| +| Every `informs` edge declared on an upstream artifact has a matching `informed_by` entry on the downstream artifact (or a documented canonical-single-place choice) | A Phase 3 walk finds an upstream `informs` with no corresponding downstream `informed_by` | +| `monitoring-setup` → `metrics-dashboard` is preserved as a valid cross-phase informs edge | A review flags the edge as invalid because it crosses a phase boundary | +| Schema validator extension for edge symmetry is filed as a follow-up Phase 1B concern | Edge symmetry drift recurs and is not surfaced mechanically | + +## References + +- [PRD](../../01-frame/prd.md) +- [Plan: 2026-05-30 Artifact Types and Concerns Audit](../plan-2026-05-30-artifact-types-and-concerns-audit.md) +- ADR-004: Dependencies Encoding diff --git a/docs/helix/02-design/adr/ADR-014-architecture-style-slot-shape-deferred.md b/docs/helix/02-design/adr/ADR-014-architecture-style-slot-shape-deferred.md new file mode 100644 index 00000000..3657a0f1 --- /dev/null +++ b/docs/helix/02-design/adr/ADR-014-architecture-style-slot-shape-deferred.md @@ -0,0 +1,80 @@ +# ADR-014: Deferred: architecture-style slot occupants remain four separate concerns + +| Date | Status | Deciders | Related | Confidence | +|------|--------|----------|---------|------------| +| 2026-05-30 | Deferred | HELIX maintainers | concerns/onion-architecture, concerns/clean-architecture, concerns/hexagonal-architecture, concerns/classic-layered, ADR-006 | Medium | + +## Context + +| Aspect | Description | +|--------|-------------| +| Problem | The architecture-style slot has four occupants (`onion-architecture`, `clean-architecture`, `hexagonal-architecture`, `classic-layered`) that sustain substantial content overlap by necessity — they are mutually exclusive at composition time and share the same shape (layering, dependency direction, boundary rules). The audit asked whether they should collapse into one concern with four variants, or remain as four separate concerns. | +| Current State | Each occupant ships its own `concern.md` + `practices.md`. The orthogonality audit classified the cluster as by-design slot competition. The verbosity attributed to these concerns is largely a consequence of the concern file shape (boundary restated in `concern.md`, `practices.md` "Boundary with neighbors", and Constraints/Drift Signals/Quality Gates), which is addressed separately by the concern file shape ADR. | +| Requirements | The audit needs a recorded decision so Phase 3 can proceed without re-opening this question. The decision must respect the audit's scope (it does not own the slot model itself). | + +## Decision + +We defer the question of whether the four architecture-style concerns collapse +into one concern with four variants. For this audit, the four concerns remain +separate — each keeps its own `concern.md` + `practices.md` and follows the +concern file shape ADR so the boundary is stated once. + +The question of per-occupant concern vs one concern with variants touches the +**slot model itself**, which is broader than this audit's scope. Collapsing +the architecture-style slot now would impose a shape on the other slot +families — language-runtime (`go-std` / `python-uv` / `rust-cargo` / `scala-sbt` +/ `typescript-bun`), e2e-framework (`e2e-kind` / `e2e-playwright`), and +demo-framework (`demo-asciinema` / `demo-playwright`) — without the +methodology-level analysis that decision warrants. That analysis is a separate +follow-up audit of the slot model, not a finding of the artifact-types-and- +concerns audit. + +The plan's Open Questions section preserves the question for that follow-up. + +**Key Points**: defer — not reject | four occupants remain separate concerns +for now | the slot-model decision is its own audit | concern file shape ADR +handles the restated-boundary verbosity independently + +## Alternatives + +| Option | Pros | Cons | Evaluation | +|--------|------|------|------------| +| Collapse the four architecture-style concerns into one concern with four variants | Removes the most-visible content overlap; one canonical boundary statement | Imposes a per-slot shape decision (variant container) on language-runtime, e2e-framework, and demo-framework slots without the broader methodology analysis; reshapes the slot model from inside one audit | Rejected for now: out of scope for this audit | +| Keep four separate concerns but rewrite each to be a thin delta against an "architecture-style overview" parent concern | Reduces restatement; preserves per-occupant addressability | Introduces a parent/child shape that the slot model doesn't currently support; same methodology-shape consequence as the collapse option | Rejected for now: same scope problem | +| **Keep the four concerns separate; defer the slot-model question to a dedicated follow-up audit** | Respects audit scope; lets the concern file shape ADR address the verbosity root cause without prejudging the slot model; preserves the question for the right venue | The visible content overlap persists until the slot-model audit lands | **Selected: scope-respecting deferral** | + +## Consequences + +| Type | Impact | +|------|--------| +| Positive | Phase 3 of the audit does not collapse these four concerns; each keeps its own `concern.md` + `practices.md`. | +| Positive | The slot-model decision is preserved for an audit that can analyze all four slot families coherently (architecture-style, language-runtime, e2e-framework, demo-framework). | +| Positive | The concern file shape ADR addresses the restated-boundary verbosity independently of the slot-model question, so the architecture-style concerns benefit from that fix without prejudging this one. | +| Negative | The visible content overlap among onion / clean / hexagonal / classic-layered persists until the slot-model audit lands. | +| Negative | Readers comparing the four occupants still navigate four separate files rather than one variant document. | +| Neutral | The architecture-style slot remains mutually exclusive at composition time; readers still pick exactly one. | + +## Risks + +| Risk | Prob | Impact | Mitigation | +|------|------|--------|------------| +| The deferred question is forgotten and the four concerns drift further apart | M | M | Plan's Open Questions section records the question; the follow-up slot-model audit picks it up | +| Editors trim restatement per-file in ways that re-introduce drift between the four occupants | M | M | The concern file shape ADR enforces "boundary stated once"; per-occupant trims reference-not-restate the slot's shared shape | +| The slot-model audit collapses the architecture-style slot later and invalidates per-occupant edits made in the meantime | L | M | Phase 3 edits to these four concerns are limited to file-shape compliance, not structural rewrites | + +## Validation + +| Success Metric | Review Trigger | +|----------------|----------------| +| The four architecture-style concerns each comply with the concern file shape ADR (boundary stated once) | A subsequent audit finds restated boundary prose across the four occupants | +| The plan's Open Questions section preserves the slot-model question for a follow-up audit | The question is silently dropped without an audit landing | +| Phase 3 does not include structural rewrites of these four concerns | A Phase 3 commit collapses or restructures the architecture-style slot | + +## References + +- Plan: docs/helix/02-design/plan-2026-05-30-artifact-types-and-concerns-audit.md (Open Questions item 7) +- ADR-006: Concern boundary lives once +- workflows/concerns/onion-architecture/ +- workflows/concerns/clean-architecture/ +- workflows/concerns/hexagonal-architecture/ +- workflows/concerns/classic-layered/ diff --git a/docs/resources/databricks-platform-substitution.md b/docs/resources/databricks-platform-substitution.md new file mode 100644 index 00000000..ea685647 --- /dev/null +++ b/docs/resources/databricks-platform-substitution.md @@ -0,0 +1,51 @@ +# Databricks Platform Substitution Reference + +If you are adopting HELIX data artifacts (`data-prd`, `data-architecture`, +`data-quality-expectations`) on a platform other than Databricks, substitute +the platform-specific terms below. The artifact templates and prompts stay +the same; only the named platform features change. + +## Catalog / Governance + +| Databricks Concept | Snowflake Equivalent | BigQuery Equivalent | On-Prem / Other | +|---|---|---|---| +| Unity Catalog (UC) hierarchy (metastore → catalog → schema → volume/table) | Database + Schema + Table | Project + Dataset + Table | Database + Schema | +| UC fine-grained access (row filters, column masks) | Row-access policies + masking policies | Authorized views + column-level security | Database views, RLS extensions | +| UC lineage | Object dependencies + Account Usage | Data lineage in Dataplex | Manual lineage in dbt/Marquez | + +## Ingestion / Streaming + +| Databricks Concept | Snowflake Equivalent | BigQuery Equivalent | On-Prem / Other | +|---|---|---|---| +| Auto Loader (incremental cloud-file ingestion) | Snowpipe or native connectors | Dataflow, BigQuery Connector Hub, Streaming Inserts | Apache NiFi, Kafka connectors | +| Streaming Tables (declarative streaming with EXPECT) | Stream-triggered materialized views + native checks | Dataflow with Beam assertions | Apache Flink with custom state management | +| Spark Structured Streaming | Snowflake streams + tasks | Dataflow | Apache Spark / Flink | + +## Orchestration / Pipelines + +| Databricks Concept | Snowflake Equivalent | BigQuery Equivalent | On-Prem / Other | +|---|---|---|---| +| Databricks Workflows / Jobs | Snowflake Tasks | Cloud Composer (Airflow) or Cloud Workflows | Apache Airflow, Dagster, dbt Cloud | +| Delta Live Tables / SDP pipeline orchestration | Dynamic Tables | Dataform | dbt, Dagster assets | + +## Quality / Contracts + +| Databricks Concept | Snowflake Equivalent | BigQuery Equivalent | On-Prem / Other | +|---|---|---|---| +| SDP `EXPECT ... ON VIOLATION ...` | Data Quality checks + Task error handling | BigQuery Data Quality API + Cloud Workflows | dbt tests, Great Expectations, custom assertions | +| SDP Genie test generation | dbt auto-generate tests from table samples | BigQuery Data Catalog insights | dbt, custom metadata scanning | +| Lakeview dashboards (quality monitoring) | Snowflake Dashboards | Looker, Data Studio | Grafana, custom dashboards | + +## Storage / Formats + +| Databricks Concept | Snowflake Equivalent | BigQuery Equivalent | On-Prem / Other | +|---|---|---|---| +| Delta Lake format | Iceberg or proprietary formats | Native BigQuery tables | Apache Parquet, Iceberg, Hudi | +| Medallion layers (Bronze / Silver / Gold) | Same pattern applies universally | Same pattern applies universally | Same pattern applies universally | + +## Compute / Cost + +| Databricks Concept | Snowflake Equivalent | BigQuery Equivalent | On-Prem / Other | +|---|---|---|---| +| DBU budget estimation | Credit consumption | On-demand or flat-rate pricing | Compute resource allocation | +| All-purpose / Jobs / Serverless SQL compute tiers | Warehouses (XS through 6XL) | On-demand vs reservation slots | Cluster sizing | diff --git a/docs/website/content/artifact-types/deploy/monitoring-setup.md b/docs/website/content/artifact-types/deploy/monitoring-setup.md index 970854b3..5303e841 100644 --- a/docs/website/content/artifact-types/deploy/monitoring-setup.md +++ b/docs/website/content/artifact-types/deploy/monitoring-setup.md @@ -114,20 +114,13 @@ ddx: - Import/review SLO burn above 25% in a day triggers rollout hold and owner review. -## Incident Response +## Alert Routing -### Response Entry Points -- Primary owner or schedule: DepositMatch on-call engineer. -- Secondary owner or schedule: platform lead. -- Immediate containment actions: disable import, pause review decisions, or - disable review-log export through feature flags. -- Existing runbook links: `docs/helix/05-deploy/runbook.md`. - -### Routing -- Primary: application on-call. +- Primary: application on-call (DepositMatch on-call engineer). - Secondary: platform lead. - Escalation: product owner for pilot-impact decisions; security lead for data exposure or control failure. +- Runbook entry point: `docs/helix/05-deploy/runbook.md`. `````` @@ -143,6 +136,6 @@ ddx: InformsDeployment Verification
Metrics Dashboard HELIX documentsdocs/helix/05-deploy/monitoring-setup.md Generation prompt
Show the full generation prompt
# Monitoring Setup Generation Prompt

Create a concise, service-specific monitoring setup for this deployment.

## Reference Anchors

Use these local resources as grounding:

- `docs/resources/google-sre-monitoring-distributed-systems.md` grounds
  operational signals, dashboards, alerting, and the four golden signals.

## Focus
- Include only the metrics, logs, alerts, dashboards, and tracing needed to operate this service.
- Define measurable thresholds, routing, and escalation where they matter.
- Connect health checks and SLOs to rollout safety and rollback decisions.
- Avoid generic observability boilerplate that does not change operator behavior.
- Separate user-impacting alerts from dashboards that are only for diagnosis.

## Completion Criteria
- Core metrics and dashboards are defined.
- Alert thresholds and routing are explicit.
- Logging, tracing, and health-check expectations are clear.
- The setup is specific enough to support deployment and incident response.
- Every page-worthy alert has an operator action or runbook entrypoint.
-Template
Show the template structure
---
ddx:
  id: monitoring-setup
---

# Monitoring Setup

## Service Summary

- Service: [service name]
- Signals that matter most: [availability, latency, throughput, errors, business KPIs]

## Metrics Collection

| Category | Metrics | Notes |
|----------|---------|-------|
| Application | [Latency, throughput, error rate] | [By endpoint or workload if needed] |
| System | [CPU, memory, disk, network] | [Only what affects service health] |
| Business | [KPI names] | [Only if operationally relevant] |
| Custom | [Metric name] | [Why it matters] |

## Alerting Rules

| Alert | Condition | Action |
|-------|-----------|--------|
| [Critical alert] | [Page threshold] | [Page path] |
| [Warning alert] | [Notification threshold] | [Notify path] |

## Dashboards

| Dashboard | Must Show |
|-----------|-----------|
| Operations | [Health, latency, errors, dependency status] |
| Business | [Adoption or outcome metrics] |
| Technical | [Resource use, queues, caches, jobs] |

## Logs and Tracing

### Logging
- Required fields: `timestamp`, `level`, `service`, `trace_id`, `message`
- Retention: [hot and cold retention]

### Tracing
- Critical journeys: [What must be traceable]
- Sampling: [Baseline and overrides]

## Health Checks

| Check | Endpoint or Mechanism | What It Verifies |
|-------|-----------------------|------------------|
| Liveness | `GET /health/live` | [Process is running] |
| Readiness | `GET /health/ready` | [Dependencies, capacity, migrations] |

## SLI/SLO Tracking

| Indicator | SLI | SLO |
|-----------|-----|-----|
| Availability | [Formula] | [Target] |
| Latency | [Formula] | [Target] |
| Quality | [Formula] | [Target] |

### Error Budget
- [Budget and escalation thresholds]

## Incident Response

### Response Entry Points
- Primary owner or schedule: [Who handles the first page]
- Secondary owner or schedule: [Who backs up the first responder]
- Immediate containment actions: [Safe first actions before deeper procedures exist]
- Existing runbook links (optional): [Add links once a runbook exists]

### Routing
- Primary: [Schedule]
- Secondary: [Schedule]
- Escalation: [Manager]
+Template
Show the template structure
---
ddx:
  id: monitoring-setup
---

# Monitoring Setup

## Service Summary

- Service: [service name]
- Signals that matter most: [availability, latency, throughput, errors, business KPIs]

## Metrics Collection

| Category | Metrics | Notes |
|----------|---------|-------|
| Application | [Latency, throughput, error rate] | [By endpoint or workload if needed] |
| System | [CPU, memory, disk, network] | [Only what affects service health] |
| Business | [KPI names] | [Only if operationally relevant] |
| Custom | [Metric name] | [Why it matters] |

## Alerting Rules

| Alert | Condition | Action |
|-------|-----------|--------|
| [Critical alert] | [Page threshold] | [Page path] |
| [Warning alert] | [Notification threshold] | [Notify path] |

## Dashboards

| Dashboard | Must Show |
|-----------|-----------|
| Operations | [Health, latency, errors, dependency status] |
| Business | [Adoption or outcome metrics] |
| Technical | [Resource use, queues, caches, jobs] |

## Logs and Tracing

### Logging
- Required fields: `timestamp`, `level`, `service`, `trace_id`, `message`
- Retention: [hot and cold retention]

### Tracing
- Critical journeys: [What must be traceable]
- Sampling: [Baseline and overrides]

## Health Checks

| Check | Endpoint or Mechanism | What It Verifies |
|-------|-----------------------|------------------|
| Liveness | `GET /health/live` | [Process is running] |
| Readiness | `GET /health/ready` | [Dependencies, capacity, migrations] |

## SLI/SLO Tracking

| Indicator | SLI | SLO |
|-----------|-----|-----|
| Availability | [Formula] | [Target] |
| Latency | [Formula] | [Target] |
| Quality | [Formula] | [Target] |

### Error Budget
- [Budget and escalation thresholds]

## Alert Routing

- Primary: [Schedule receiving first page]
- Secondary: [Backup schedule]
- Escalation: [Manager or coordinator path]
- Runbook entry point: [Link to runbook once it exists]
diff --git a/docs/website/content/artifact-types/design/architecture.md b/docs/website/content/artifact-types/design/architecture.md index 16e48f6f..a137bc5c 100644 --- a/docs/website/content/artifact-types/design/architecture.md +++ b/docs/website/content/artifact-types/design/architecture.md @@ -15,10 +15,8 @@ activity. Its unique job is to describe the durable system shape: boundaries, containers, externally visible integrations, deployment topology, critical data flows, quality attributes, and structural tradeoffs. -Architecture is not a feature solution design, story technical design, ADR, or -runbook. ADRs record individual decisions. Solution and technical designs apply -the architecture to narrower scopes. Operational artifacts inherit deployment -and quality constraints from architecture. +For what belongs at this level versus Solution Design and Technical Design, see +the zoom-stack matrix in `workflows/activities/02-design/README.md`. ## Example @@ -201,7 +199,7 @@ sequenceDiagram InformsSolution Design
Technical Design
ADR
Contract
Data Design
Security Architecture Referenced bySolution Design
Technical Design
Deployment Checklist
Runbook
Monitoring Setup HELIX documentsdocs/helix/02-design/architecture.md -Generation prompt
Show the full generation prompt
# Architecture Documentation Generation Prompt

Document the architecture views that the team actually needs to build, review,
operate, and evolve the system.

## Purpose

Architecture is the **highest-authority structural artifact** in the Design
activity. Its unique job is to describe the durable system shape: boundaries,
containers, externally visible integrations, deployment topology, critical data
flows, quality attributes, and structural tradeoffs.

Architecture is not a feature solution design, story technical design, ADR, or
runbook. ADRs record individual decisions. Solution and technical designs apply
the architecture to narrower scopes. Operational artifacts inherit deployment
and quality constraints from architecture.

## Reference Anchors

Use these local resource summaries as grounding:

- `docs/resources/c4-model.md` grounds context, container, component, and
  deployment views as audience-specific structural diagrams.
- `docs/resources/sei-quality-attribute-scenarios.md` grounds measurable
  quality attributes and tradeoff reasoning.
- `docs/resources/microsoft-azure-well-architected-framework.md` grounds
  cross-cutting reliability, security, operations, performance, and cost
  tradeoffs.

## Focus
- Include only the C4 views that add information; omit empty or duplicate views.
- Keep boundaries, deployment shape, data flow, and quality attributes visible.
- Annotate major tradeoffs or constraints directly on the relevant view or summary.
- Remove generic architecture commentary.

## Boundary Test

| If you are writing... | Put it in... |
|---|---|
| Product behavior, priority, or success metrics | PRD or Feature Specification |
| One structural decision with alternatives and consequences | ADR |
| Feature-specific system design | Solution Design |
| Story-level implementation plan | Technical Design |
| Endpoint schemas or message formats | Contract |
| Deployment execution steps, rollback, or release readiness | Deployment Checklist or Runbook |

## Completion Criteria
- The views are understandable at a glance.
- Key boundaries and tradeoffs are visible.
- The document stays implementation-relevant.
- Containers name real technologies, not generic roles.
- Quality attributes have measurable targets and verification methods.
- Deployment names actual infrastructure shape, scaling, and backup approach.
- Major decisions link to ADRs or include an inline decision note.
+Generation prompt
Show the full generation prompt
# Architecture Documentation Generation Prompt

Document the architecture views that the team actually needs to build, review,
operate, and evolve the system.

## Purpose

Architecture is the **highest-authority structural artifact** in the Design
activity. Its unique job is to describe the durable system shape: boundaries,
containers, externally visible integrations, deployment topology, critical data
flows, quality attributes, and structural tradeoffs.

For what belongs at this level versus Solution Design and Technical Design, see
the zoom-stack matrix in `workflows/activities/02-design/README.md`.

## Reference Anchors

Use these local resource summaries as grounding:

- `docs/resources/c4-model.md` grounds context, container, component, and
  deployment views as audience-specific structural diagrams.
- `docs/resources/sei-quality-attribute-scenarios.md` grounds measurable
  quality attributes and tradeoff reasoning.
- `docs/resources/microsoft-azure-well-architected-framework.md` grounds
  cross-cutting reliability, security, operations, performance, and cost
  tradeoffs.

## Focus
- Include only the C4 views that add information; omit empty or duplicate views.
- Keep boundaries, deployment shape, data flow, and quality attributes visible.
- Annotate major tradeoffs or constraints directly on the relevant view or summary.
- Remove generic architecture commentary.

## Boundary Test

See the zoom-stack matrix in `workflows/activities/02-design/README.md` for
which decisions belong at the system, feature, and story levels.

## Completion Criteria
- The views are understandable at a glance.
- Key boundaries and tradeoffs are visible.
- The document stays implementation-relevant.
- Containers name real technologies, not generic roles.
- Quality attributes have measurable targets and verification methods.
- Deployment names actual infrastructure shape, scaling, and backup approach.
- Major decisions link to ADRs or include an inline decision note.
Template
Show the template structure
---
ddx:
  id: architecture
---

# Architecture

## Scope

[What system this architecture covers, what is deliberately outside the
architecture boundary, and which PRD/features/user journeys drive the design.]

## Level 1: System Context

| Element | Type | Purpose | Protocol |
|---------|------|---------|----------|
| [User/System] | User/External | [Interaction] | [HTTP/API/etc] |

```mermaid
graph TB
    %% Add users, system, and external dependencies
```

## Level 2: Container Diagram

| Container | Technology | Responsibilities | Communication |
|-----------|------------|------------------|---------------|
| [Name] | [Stack] | [What it does] | [Protocol/Format] |

```mermaid
graph TB
    %% Add containers: Web, API, DB, Queue, Worker, external systems
```

## Level 3: Component Diagram

Include only when a container needs internal structure to support downstream
design and review. Omit this section when container responsibilities are enough.

| Component | Container | Purpose | Notes |
|-----------|-----------|---------|-------|
| [Name] | [Container] | [Responsibility] | [Key decisions] |

```mermaid
graph TB
    %% Optional: add components inside the container that needs detail
```

## Deployment

| Component | Infrastructure | Instances | Scaling | Backup / Recovery |
|-----------|----------------|-----------|---------|-------------------|
| [Name] | [Cloud/service/runtime] | [Count] | [Trigger or limit] | [Backup/failover path] |

```mermaid
graph TB
    %% Add actual deployment topology
```

## Data Flow

```mermaid
sequenceDiagram
    %% Add sequence for the most important user journey or operational flow
```

## Quality Attributes

| Attribute | Target | Strategy | Verification |
|-----------|--------|----------|--------------|
| Availability | [Target] | [How architecture supports it] | [How checked] |
| Performance | [Target] | [How architecture supports it] | [How checked] |
| Security | [Target] | [Controls/boundaries] | [How checked] |
| Disaster Recovery | RTO: [target] / RPO: [target] | [Backup/failover strategy] | [How checked] |

## Decisions and Tradeoffs

| Decision | Status | Rationale | Follow-up |
|----------|--------|-----------|-----------|
| [ADR-NNN or inline decision] | [Accepted/Proposed] | [Why this shape wins] | [ADR, spike, or none] |
diff --git a/docs/website/content/artifact-types/design/data-architecture.md b/docs/website/content/artifact-types/design/data-architecture.md index a06bfffb..07a81eee 100644 --- a/docs/website/content/artifact-types/design/data-architecture.md +++ b/docs/website/content/artifact-types/design/data-architecture.md @@ -17,7 +17,7 @@ transformation patterns, governance boundaries, quality gates, and critical performance or cost tradeoffs. Data Architecture is not a data model (captured in Data Design), implementation plan, -or ADR. It is the bridge between Data PRD (requirements) and implementation: "given +or ADR. It is the bridge between PRD (kind: data) (requirements) and implementation: "given these requirements, here is how the pipeline is structured." ## Example @@ -29,8 +29,9 @@ these requirements, here is how the pipeline is structured." --- ddx: id: example.data-architecture.customer-360 - depends_on: - - example.data-prd.customer-360 + # Previous depends_on: example.data-prd.customer-360 — dropped when + # data-prd collapsed into prd as kind: data variant (ADR-008). No + # equivalent example.prd.customer-360 is yet published. --- # Data Architecture: Customer-360 Analytics @@ -197,7 +198,7 @@ sequenceDiagram EnablesNone InformsData Quality Expectations
Technical Design
Solution Design Referenced byData Quality Expectations
Implementation Plan
Runbook -Generation prompt
Show the full generation prompt
# Data Architecture Generation Prompt

Document the data pipeline architecture that the team needs to build, review,
operate, and evolve the data product.

## Purpose

Data Architecture is the **highest-authority structural artifact for data pipeline
design** in the Design activity. Its unique job is to describe the durable pipeline
shape: ingestion patterns, medallion layer topology, streaming vs. batch semantics,
transformation patterns, governance boundaries, quality gates, and critical
performance or cost tradeoffs.

Data Architecture is not a data model (captured in Data Design), implementation plan,
or ADR. It is the bridge between Data PRD (requirements) and implementation: "given
these requirements, here is how the pipeline is structured."

## Reference Anchors

Use these local resource summaries as grounding:

- `docs/resources/databricks-lakehouse-medallion-architecture.md` grounds
  medallion topology (Bronze/Silver/Gold layer responsibilities, transformations,
  and quality gates).
- `docs/resources/databricks-auto-loader.md` grounds cloud-native ingestion
  patterns for incremental, scalable, schema-aware source connectors.
- `docs/resources/databricks-streaming-tables.md` grounds declarative streaming
  and materialized views for real-time transformations and quality enforcement.
- `docs/resources/databricks-sdp.md` grounds SDP lineage, governance, and
  quality-first design through `EXPECT ... ON VIOLATION ...` clauses and
  contract-driven pipeline composition.

## Focus

- Sketch the medallion layer flow: what lands in Bronze, what transformations
  happen in Silver, what business tables live in Gold.
- Name ingestion patterns (Auto Loader, Streaming Tables, batched SQL, CDC) and
  why each is used for its source.
- Document transformation semantics: idempotence, exactly-once vs. at-least-once,
  stateful operations, and how schema evolution is handled.
- Specify governance and quality checkpoints: where data is validated, which
  layers enforce which contracts, and how SLA compliance is monitored.
- Call out critical performance or cost tradeoffs: partitioning strategy,
  clustering, retention policy, incremental refresh vs. full rebuild.

## Role Boundary

Data Architecture describes pipeline topology and data flow, not the detailed
data model (Data Design), not implementation sequences (Implementation Plan),
and not individual quality checks (Data Quality Expectations).

**Databricks Platform Substitution:** If you are adopting this on another data
platform, substitute as follows:

| Databricks Concept | Snowflake Equivalent | BigQuery Equivalent | On-Prem / Other |
|---|---|---|---|
| Medallion layers (Bronze/Silver/Gold) | Same pattern applies universally | Same pattern applies universally | Same pattern applies universally |
| Auto Loader | Snowpipe or native connectors | Dataflow, BigQuery Connector Hub | Apache NiFi, Kafka connectors |
| Streaming Tables with `EXPECT` clauses | Stream-triggered materialized views + native checks | Dataflow with Beam assertions | Apache Flink with custom state management |
| Databricks Jobs for orchestration | Snowflake Tasks | Cloud Composer (Airflow) or Cloud Workflows | Apache Airflow, Dagster, dbt Cloud |
| SDP `EXPECT ... ON VIOLATION ...` | Data Quality checks + Task error handling | BigQuery Data Quality API + Cloud Workflows | dbt tests, Great Expectations, custom assertions |
| Delta Lake format | Iceberg or proprietary formats | Native BigQuery tables | Apache Parquet, Iceberg, Hudi |

## Completion Criteria

- Medallion layer diagram or description is clear (what lands where, why).
- Each layer's transformation responsibilities are explicit.
- Ingestion patterns name actual technologies and explain why each is used.
- Quality gates are named (where validation happens, what contracts are
  enforced).
- Performance/cost tradeoffs are visible (partitioning, clustering, retention,
  refresh strategy).
- Deployment topology is concrete (number of clusters, auto-scaling, failover).
- Major decisions link to Data PRD requirements or include inline rationale.
-Template
Show the template structure
---
ddx:
  id: data-architecture
---

# Data Architecture

## Overview

[Describe the data product being architected, the business problem it solves, and the system context. Name the key data flows and Databricks platform fit. Reference the [[data-prd]] for the business requirements and success metrics this architecture must satisfy.]

### Scope

[What data flows and systems are covered by this architecture? What is deliberately outside the boundary? Which requirements from [[data-prd]] drive the design decisions?]

### System Context

| External System | Role | Protocol | Data Volume |
|-----------------|------|----------|------------|
| [e.g., Salesforce] | [Source system for customer data] | [REST API, batch export] | [M records/day] |
| [e.g., Data Warehouse] | [Consumer of aggregated data] | [Delta API, SQL] | [K rows/hour] |

```mermaid
graph TB
    A["Salesforce<br/>(Source)"] -->|REST API<br/>hourly| B["Databricks<br/>Data Platform"]
    C["Stripe<br/>(Source)"] -->|S3 batch export| B
    B -->|Consumption Layer| D["BI Tool<br/>(Looker/Tableau)"]
    B -->|ML Training| E["ML Platform"]
```

## Medallion Topology

### Architecture Pattern

[Describe the medallion layer strategy: Bronze (raw), Silver (validated), Gold (consumption). For each layer, explain the transformation scope, quality gates, and consumer responsibilities.]

### Bronze Layer (Raw Ingestion)

**Purpose**: Land source data in its native form without transformation.

**Source Integration**:
- Source System: [e.g., Salesforce]
- Integration Pattern: [Auto Loader, Streaming Tables, scheduled batch import]
- Schema Validation: [Applied at ingestion, reject if schema mismatch]
- Retention: [e.g., 7 days rolling window for cost efficiency]

**Responsibilities**:
- Ingest all records from source
- Preserve source schema exactly (no column renames or type coercion)
- Tag records with `_ingest_timestamp` and source system identifier
- Detect and quarantine records that fail schema validation

**Quality Gates**:
- All records have `_ingest_timestamp` and source system id
- No truncation of source columns
- Fail fast if source unavailable for >4 hours

### Silver Layer (Validated and Transformed)

**Purpose**: Cleansed, deduplicated, and business-logic-ready data.

**Transformations**:
- Deduplication: [Method: e.g., "keep latest by timestamp", "PK uniqueness constraint"]
- Data type coercion: [e.g., "convert dates to ISO-8601, currency to numeric"]
- Null handling: [e.g., "fill defaults, or fail if critical column null"]
- Referential integrity: [e.g., "customer_id must exist in customer dimension"]

**Join Strategy**:
| Join | Left | Right | Type | Cardinality | Latency Impact |
|------|------|-------|------|-------------|---|
| [customer enrichment] | `customer_events` | `customer_master` | [Left Outer] | [1:1] | [Add 100ms] |

**Retention**: [e.g., 90 days]

**Responsibilities**:
- Enforce UNIQUE and NOT NULL constraints per data quality expectations
- Materialize aggregations needed for Gold layer
- Document row counts and latency metrics per transformation

**Quality Gates**:
- Zero duplicates (PK uniqueness)
- ≥99% NOT NULL on critical columns
- Row count reconciliation with source ±0.1%

### Gold Layer (Consumption)

**Purpose**: Business-ready tables optimized for consumer queries.

**Consumption Tables**:
| Table | Use Case | Consumers | Freshness | Aggregation |
|-------|----------|-----------|-----------|------------|
| `customer_360` | Single customer view, 360-degree profile | Analytics, ML | Hourly | None (deduped Silver) |
| `daily_sales_summary` | Daily revenue by product | Finance, BI | Daily | SUM(amount) by product_date |

**Optimization Strategy**:
- Partitioning: [e.g., "by date for date-range queries"]
- Z-order: [e.g., "on customer_id, product_id for filter selectivity"]
- Caching: [Materialized views for slow queries]

**Retention**: [e.g., 2 years for compliance and analytics]

**Responsibilities**:
- Keep Gold current with Silver ingestion cadence
- Publish schema and SLA via UC catalog comments
- Monitor query performance; alert if p95 latency > SLA

**Quality Gates**:
- Sums reconcile between Silver aggregates and Gold tables ±0.01%
- All customer IDs in Gold exist in Silver Silver
- Latency ≤SLA (e.g., ≤5 seconds for 90th percentile query)

## Data Flow

[Describe how data moves through the medallion layers. Clarify ingestion frequency, transformation latency, and refresh strategy.]

### Ingestion Flow

```mermaid
graph LR
    A["Salesforce API"] -->|Auto Loader<br/>every 15 min| B["Bronze<br/>raw_customers"]
    B -->|PySpark Job<br/>dedup + validate| C["Silver<br/>customers_validated"]
    C -->|SQL notebook<br/>daily 9am UTC| D["Gold<br/>customer_360"]
    D -->|Published to UC| E["Consumers<br/>BI, ML"]
```

### Incremental vs Full Refresh

- **Bronze**: [Incremental via change data capture, or full reload?]
- **Silver**: [Incremental updates on specific key columns, or full recalc?]
- **Gold**: [Append-only fact tables, or snapshot updates?]

## Processing Semantics

### Streaming vs Batch Decision

| Layer | Strategy | Rationale | SLA Implication |
|-------|----------|-----------|-----------------|
| Bronze | [Streaming / Batch / Incremental Batch] | [Why this choice?] | [Freshness achieved] |
| Silver | [Streaming / Batch / Incremental Batch] | [Why this choice?] | [Freshness achieved] |
| Gold | [Streaming / Batch / Incremental Batch] | [Why this choice?] | [Freshness achieved] |

### Processing Framework

- **Framework**: [Databricks SQL, PySpark, dbt on Databricks, Streaming Tables]
- **Orchestration**: [Databricks Workflows, Airflow, dbt Cloud]
- **Failure Handling**: [Retry policy, dead-letter queue, manual intervention?]

### Latency and Throughput Targets

| Stage | Target Latency | Target Throughput | Constraint |
|-------|-----------------|-------------------|-----------|
| Salesforce → Bronze | ≤15 minutes | 1M records/day | API rate limit: 100 req/min |
| Bronze → Silver | ≤30 minutes | 1M records/day | PySpark cluster size |
| Silver → Gold | ≤2 hours | 100K aggregates/day | SQL query complexity |

## Quality Contracts

[Define expectations as testable EXPECT clauses per Databricks SDP. Each expectation binds the architecture: data violating it is rejected before reaching consumers.]

### Bronze Layer Expectations

```sql
-- Raw data must match source schema exactly
EXPECT TABLE raw_customers (
  customer_id NOT NULL,
  email NOT NULL,
  created_at TIMESTAMP NOT NULL,
  _ingest_timestamp TIMESTAMP NOT NULL GENERATED ALWAYS AS (CURRENT_TIMESTAMP())
);
```

- **Expectation**: All records ingested within 15 min of source commit
- **Violation**: Alert; do not advance to Silver
- **SLA**: Detect within 5 minutes

### Silver Layer Expectations

```sql
-- Customers must be unique per customer_id, deduplicated by latest timestamp
EXPECT TABLE customers_validated (
  customer_id NOT NULL,
  PRIMARY KEY (customer_id),
  UNIQUE (customer_id)
);

-- Customer email addresses must be normalized and not null
EXPECT TABLE customers_validated (
  email NOT NULL LIKE '%@%.%'
);
```

- **Expectation**: ≥99% NOT NULL on `email` and `phone`
- **Expectation**: Zero duplicate customers (PK uniqueness)
- **Violation**: Quarantine; manual review before advancing to Gold

### Gold Layer Expectations

```sql
-- Customer 360 must be current within freshness SLA (1 hour)
EXPECT TABLE customer_360 (
  MAX(_modified_at) >= CURRENT_TIMESTAMP() - INTERVAL 1 HOUR
);

-- Revenue aggregates must reconcile with Silver within 0.01%
EXPECT TABLE daily_sales_summary
  CHECK (
    SELECT SUM(amount) FROM daily_sales_summary 
    IS WITHIN 0.01% OF 
    SELECT SUM(amount) FROM customers_validated
  );
```

- **Expectation**: Customer 360 is no older than 1 hour
- **Expectation**: Daily sales sums reconcile with Silver ±$0.01 per order
- **Violation**: Reject; roll back to prior snapshot; alert on-call

### Cross-Layer Data Contracts

| Contract | Assertion | If Violated |
|----------|-----------|------------|
| [Bronze → Silver row count] | [Silver row count ≤ Bronze + 10% for dedup] | [Alert + manual audit] |
| [Silver → Gold cardinality] | [Gold unique customers = Silver unique customers] | [Reject until reconciled] |
| [Foreign key integrity] | [All orders.customer_id exist in customer_360] | [Quarantine order; alert] |

## Governance and Access Control

### Identity and Access Management

| Role | Catalog | Schema | Table | Permissions |
|------|---------|--------|-------|------------|
| Data Engineer | `prod` | `customer_360` | All | READ, MODIFY, EXECUTE |
| Analytics Lead | `prod` | `customer_360` | `customer_360`, `daily_sales_summary` (Gold only) | SELECT |
| Finance Team | `prod` | `customer_360` | `daily_sales_summary` | SELECT (date ≥ 2 years ago) |

### Data Classification and Retention

| Table | Classification | Sensitive Columns | Retention | Masked For |
|-------|------------------|------------------|-----------|-----------|
| `raw_customers` | Internal | `ssn`, `credit_card` | 7 days | [Not masked; only admins see] |
| `customers_validated` | Internal | `email`, `phone` | 90 days | [Non-PII audiences mask these] |
| `customer_360` | Internal | `email` | 2 years | [Mask for BI tool] |

### Fine-Grained Access Control

- **Row-Level Security**: [Do analytics users see only their region's data? Document the policy.]
- **Column-Level Security**: [Which sensitive columns are masked for which roles?]
- **Dynamic Views**: [Use UC masking functions to redact PII per caller role?]

## Databricks Platform Design

### Catalog Organization

```
prod (Catalog)
├── customer_360 (Schema)
│   ├── raw_customers (Bronze table)
│   ├── customers_validated (Silver table)
│   ├── customer_360 (Gold table)
│   └── daily_sales_summary (Gold aggregate)
├── metadata (Schema)
│   ├── pipeline_runs (audit log)
│   └── quality_metrics (expectations results)
```

### Compute Strategy

| Workload | Compute Tier | Cluster Size | DBU Budget | Rationale |
|----------|--------------|--------------|------------|-----------|
| Bronze ingestion (streaming) | All-purpose | 4 workers (i3.xlarge) | 8 DBU/hour | Continuous streaming |
| Silver transformation (batch) | Jobs | 8 workers (i3.2xlarge) | 16 DBU/run | Scheduled nightly |
| Gold aggregation (query) | Serverless SQL | N/A | ~5 DBU/query | Ad-hoc BI queries |

**Cost Optimization**:
- Spot instances for non-critical workloads: 30% savings
- Auto-terminate idle clusters: reduce waste
- Partition pruning and z-order for faster queries: reduce scans

### Storage Strategy

| Layer | Format | Compression | Optimization |
|-------|--------|-------------|--------------|
| Bronze | Delta | Snappy | Partitioned by date (rolling 7-day window) |
| Silver | Delta | Snappy | Z-ordered by customer_id, product_id |
| Gold | Delta | Snappy | Partitioned by date; cached for BI queries |

**Storage Sizing**: ~500 GB Bronze, ~50 GB Silver, ~5 GB Gold (monthly) → ~$25/month at $0.023/GB/month

### Databricks Features in Use

| Feature | Use Case | Configuration |
|---------|----------|---------------|
| Auto Loader | Salesforce → Bronze incremental ingestion | Cloud file location: S3, Schema inference enabled |
| Streaming Tables | Bronze → Silver continuous transformation | Trigger: arrival, 30-second max latency |
| Delta Live Tables | End-to-end medallion pipeline orchestration | Refresh: hourly for Bronze/Silver, daily for Gold |
| Unity Catalog | Governance, lineage, fine-grained access | Enable open sharing across teams |

## Decisions and Tradeoffs

### Key Architecture Decisions

| Decision | Choice | Rationale | Alternative Considered | Consequence |
|----------|--------|-----------|------------------------|------------|
| [Medallion layers] | [Bronze/Silver/Gold] | [Standard Databricks pattern for quality gates] | [Flat schema, 2-layer] | [Slower queries if using Bronze directly; quality risk] |
| [Streaming vs Batch for Silver] | [Batch nightly] | [Simplicity, cost] | [Streaming] | [1-2 hour stale window; acceptable for reporting] |
| [Compute tier for Gold queries] | [Serverless SQL] | [Cost & elasticity; no cluster mgmt] | [All-purpose cluster] | [Higher per-query cost; simpler ops] |

### Performance vs Cost Tradeoffs

- **Real-time ingestion** (streaming Bronze): Freshness ≤5 min, but 40% higher DBU cost
- **Materialized Gold aggregates**: Faster queries (100ms), but 20% higher storage cost
- **Spot instances for Silver jobs**: 30% cheaper, but risk of interruption (retry handles it)

### Known Risks and Mitigations

| Risk | Mitigation |
|------|-----------|
| [Salesforce API rate limit causes ingestion backlog] | [Implement exponential backoff; buffer in queue; alert if >1 hour backlog] |
| [PII in Bronze accessible to all engineers] | [Mask sensitive columns in published views; audit access logs] |
| [Schema drift in source breaks ingestion] | [Schema registry; auto-detect new columns; manual approval before Silver] |

---

## Review Checklist

Use this checklist during review to validate that the data architecture is complete and ready for implementation:

- [ ] **Scope** clearly states which data flows are included and which are out of bounds
- [ ] **Medallion Topology** defines Bronze, Silver, Gold layer purposes and transformation rules
- [ ] **Data Flow diagrams** (mermaid) show how data moves through layers and to consumers
- [ ] **Processing Semantics** explicitly state streaming vs batch for each layer with latency targets
- [ ] **Quality Contracts** are written as executable EXPECT clauses (SDP, dbt, SQL constraints)
- [ ] **Failure Handling** specifies what happens when an expectation fails (alert, reject, quarantine)
- [ ] **Access Control** model covers identity, row-level, column-level, and sensitive data masking
- [ ] **Databricks Platform Design** names catalog, schema, compute tier, storage strategy, and DBU budget
- [ ] **Decisions and Tradeoffs** document key choices with rationale and alternatives considered
- [ ] **Cross-layer data contracts** are defined (sums reconcile, cardinality stable, no orphans)
- [ ] **SLA per layer** is documented (freshness, latency, availability targets)
- [ ] No `[TBD]`, `[TODO]`, or `[NEEDS CLARIFICATION]` markers remain
- [ ] Architecture aligns with requirements in [[data-prd]] and can satisfy success metrics
- [ ] References link to [[data-quality-expectations]] for detailed EXPECT clauses per layer
+Generation prompt
Show the full generation prompt
# Data Architecture Generation Prompt

Document the data pipeline architecture that the team needs to build, review,
operate, and evolve the data product.

## Purpose

Data Architecture is the **highest-authority structural artifact for data pipeline
design** in the Design activity. Its unique job is to describe the durable pipeline
shape: ingestion patterns, medallion layer topology, streaming vs. batch semantics,
transformation patterns, governance boundaries, quality gates, and critical
performance or cost tradeoffs.

Data Architecture is not a data model (captured in Data Design), implementation plan,
or ADR. It is the bridge between PRD (kind: data) (requirements) and implementation: "given
these requirements, here is how the pipeline is structured."

## Reference Anchors

Use these local resource summaries as grounding:

- `docs/resources/databricks-lakehouse-medallion-architecture.md` grounds
  medallion topology (Bronze/Silver/Gold layer responsibilities, transformations,
  and quality gates).
- `docs/resources/databricks-auto-loader.md` grounds cloud-native ingestion
  patterns for incremental, scalable, schema-aware source connectors.
- `docs/resources/databricks-streaming-tables.md` grounds declarative streaming
  and materialized views for real-time transformations and quality enforcement.
- `docs/resources/databricks-sdp.md` grounds SDP lineage, governance, and
  quality-first design through `EXPECT ... ON VIOLATION ...` clauses and
  contract-driven pipeline composition.

## Focus

- Sketch the medallion layer flow: what lands in Bronze, what transformations
  happen in Silver, what business tables live in Gold.
- Name ingestion patterns (Auto Loader, Streaming Tables, batched SQL, CDC) and
  why each is used for its source.
- Document transformation semantics: idempotence, exactly-once vs. at-least-once,
  stateful operations, and how schema evolution is handled.
- Specify governance and quality checkpoints: where data is validated, which
  layers enforce which contracts, and how SLA compliance is monitored.
- Call out critical performance or cost tradeoffs: partitioning strategy,
  clustering, retention policy, incremental refresh vs. full rebuild.

## Role Boundary

Data Architecture describes pipeline topology and data flow, not the detailed
data model (Data Design), not implementation sequences (Implementation Plan),
and not individual quality checks (Data Quality Expectations).

**Non-Databricks platforms:** see
`docs/resources/databricks-platform-substitution.md` for the equivalent
terms on Snowflake, BigQuery, and on-prem stacks. The artifact shape and
prompt stay the same.

## Completion Criteria

- Medallion layer diagram or description is clear (what lands where, why).
- Each layer's transformation responsibilities are explicit.
- Ingestion patterns name actual technologies and explain why each is used.
- Quality gates are named (where validation happens, what contracts are
  enforced).
- Performance/cost tradeoffs are visible (partitioning, clustering, retention,
  refresh strategy).
- Deployment topology is concrete (number of clusters, auto-scaling, failover).
- Major decisions link to PRD (kind: data) requirements or include inline rationale.
+Template
Show the template structure
---
ddx:
  id: data-architecture
---

# Data Architecture

Platform- and pipeline-level shape of the data product: medallion topology,
processing-framework choices, governance model, and pipeline-level quality
contracts. Entity-level modelling (logical schema, access patterns,
constraints, migration) lives in [[data-design]].

## Overview

[Describe the data product being architected, the business problem it solves,
and the system context. Name the key data flows and platform fit. Reference
[[prd]] (kind: data) for the requirements and success metrics this architecture must
satisfy.]

### Scope

[What data flows and systems are covered. What is deliberately out of bounds.
Which requirements from [[prd]] (kind: data) drive the design decisions.]

### System Context

| External System | Role | Protocol | Data Volume |
|-----------------|------|----------|------------|
| [Source system] | [Role in the pipeline] | [API, batch export, CDC] | [Order-of-magnitude per period] |
| [Consumer system] | [How it consumes Gold] | [Delta share, SQL, API] | [Query volume] |

```mermaid
graph TB
    A["Source A"] -->|ingest| B["Data Platform"]
    C["Source B"] -->|ingest| B
    B -->|consumption layer| D["BI / Reporting"]
    B -->|feature store| E["ML Platform"]
```

## Medallion Topology

### Layer Strategy

[State the medallion strategy: Bronze (raw), Silver (validated), Gold
(consumption). For each layer, name the transformation scope, quality gates,
and consumer responsibilities. Justify the choice against [[prd]] (kind: data)
freshness and quality requirements.]

### Bronze Layer (Raw Ingestion)

- **Purpose**: Land source data in its native form without transformation.
- **Source integration pattern**: [Auto Loader, Streaming Tables, scheduled
  batch import, CDC]
- **Schema handling**: [Strict / inferred / evolution policy]
- **Retention policy**: [Rationale tied to cost and replay needs]

Responsibilities:
- Ingest all records from source.
- Preserve source schema exactly (no renames or coercion).
- Tag records with ingest timestamp and source-system identifier.
- Quarantine records that fail schema validation.

Quality gates: ingest-metadata presence, no column truncation, source
availability watchdog.

### Silver Layer (Validated and Transformed)

- **Purpose**: Cleansed, deduplicated, business-logic-ready data.
- **Deduplication strategy**: [Key + ordering rule]
- **Type coercion / null policy**: [Defaults vs reject]
- **Referential integrity**: [Which FK relationships are enforced and how]

Join strategy (pipeline-level — entity-level joins live in [[data-design]]):

| Join | Source Layers | Type | Cardinality | Latency Impact |
|------|---------------|------|-------------|----------------|
| [Logical join name] | [Left / Right] | [Inner / Outer] | [1:1 / 1:N] | [Qualitative] |

Quality gates: PK uniqueness, NOT NULL on critical columns, row-count
reconciliation with Bronze within tolerance.

### Gold Layer (Consumption)

- **Purpose**: Business-ready tables optimised for consumer queries.
- **Optimisation strategy**: [Partitioning, clustering / z-order, materialised
  views — at the pipeline level, not column-level]
- **Retention policy**: [Compliance and analytics horizon]

Consumption tables (entity definitions live in [[data-design]]):

| Table | Use Case | Consumers | Freshness Target |
|-------|----------|-----------|------------------|
| [Gold table name] | [Use case from PRD (kind: data)] | [Persona] | [Target tied to SLA] |

Quality gates: aggregate reconciliation with Silver, referential integrity
across Gold, latency within consumer SLA.

## Data Flow

[Describe how data moves through the medallion layers. Clarify ingestion
frequency, transformation latency, and refresh strategy.]

```mermaid
graph LR
    A["Source"] -->|ingest pattern| B["Bronze"]
    B -->|transform job| C["Silver"]
    C -->|aggregate job| D["Gold"]
    D -->|published| E["Consumers"]
```

### Incremental vs Full Refresh

- **Bronze**: [CDC / append / full reload — rationale]
- **Silver**: [Incremental keys / full recalc — rationale]
- **Gold**: [Append-only / snapshot / merge — rationale]

## Processing Semantics

### Streaming vs Batch Decision

| Layer | Strategy | Rationale | SLA Implication |
|-------|----------|-----------|-----------------|
| Bronze | [Streaming / Batch / Incremental] | [Why] | [Freshness achieved] |
| Silver | [Streaming / Batch / Incremental] | [Why] | [Freshness achieved] |
| Gold | [Streaming / Batch / Incremental] | [Why] | [Freshness achieved] |

### Processing Framework

- **Framework**: [Databricks SQL, PySpark, dbt, Streaming Tables, Flink, …]
- **Orchestration**: [Workflows, Airflow, dbt Cloud, Dagster, …]
- **Failure handling**: [Retry policy, dead-letter queue, manual intervention]
- **Idempotence / exactly-once posture**: [Per layer]
- **Schema evolution policy**: [Auto-add / manual approval / strict]

### Latency and Throughput Targets

| Stage | Latency Target | Throughput Target | Binding Constraint |
|-------|----------------|-------------------|--------------------|
| Source → Bronze | [From PRD (kind: data) SLA] | [Order of magnitude] | [Rate limit, API quota] |
| Bronze → Silver | [From PRD (kind: data) SLA] | [Order of magnitude] | [Compute / dedup cost] |
| Silver → Gold | [From PRD (kind: data) SLA] | [Order of magnitude] | [Query complexity] |

## Pipeline-Level Quality Contracts

[Express the contracts the *pipeline* enforces at each layer boundary.
Column-level field rules belong in [[data-quality-expectations]]; this
section names which contracts the architecture commits to enforce and
where.]

### Bronze → Silver

- **Schema contract**: [What Silver requires of Bronze]
- **Volume contract**: [Acceptable row-count delta]
- **Freshness contract**: [Max ingest lag before Silver is held]
- **Violation handling**: [Alert / hold / quarantine]

### Silver → Gold

- **Uniqueness contract**: [Which keys are unique at Gold]
- **Referential contract**: [Which FK relationships are guaranteed]
- **Aggregate-reconciliation contract**: [Sums and counts must agree within
  tolerance]
- **Violation handling**: [Reject / rollback / alert]

### Cross-Layer Contracts

| Contract | Assertion | If Violated |
|----------|-----------|-------------|
| [Row count Bronze → Silver] | [Within tolerance] | [Alert + manual audit] |
| [Cardinality Silver → Gold] | [Stable across refresh] | [Reject until reconciled] |
| [FK integrity across Gold] | [No orphans] | [Quarantine + alert] |

Detailed `EXPECT` clauses, field-level constraints, and freshness predicates
live in [[data-quality-expectations]].

## Governance and Access Control

### Identity and Access Model

| Role | Catalog Scope | Layer Access | Permissions |
|------|---------------|--------------|-------------|
| [Role from PRD (kind: data) consumers] | [Catalog / schema] | [Bronze / Silver / Gold] | [SELECT / MODIFY / EXECUTE] |

### Data Classification and Retention

| Layer | Classification | Sensitive Categories | Retention Policy | Masking Policy |
|-------|----------------|----------------------|------------------|----------------|
| Bronze | [Class] | [Categories — not specific columns; those live in data-design] | [Policy tied to compliance] | [Who sees raw] |
| Silver | [Class] | [Categories] | [Policy] | [Who sees masked vs raw] |
| Gold | [Class] | [Categories] | [Policy] | [Default masking for BI] |

### Fine-Grained Access Control

- **Row-level security**: [Tenant / region predicate — policy, not the
  predicate code, which lives in [[data-design]]]
- **Column-level security**: [Which classifications are masked for which
  roles]
- **Dynamic views**: [Masking-function strategy]

## Platform Design

### Catalog Organisation

```
[catalog]
├── [schema]
│   ├── [bronze table family]
│   ├── [silver table family]
│   └── [gold table family]
├── metadata
│   ├── pipeline_runs
│   └── quality_metrics
```

### Compute Strategy

| Workload | Compute Tier | Sizing Approach | Rationale |
|----------|--------------|-----------------|-----------|
| Bronze ingestion | [Tier] | [Auto-scale bounds / fixed] | [Continuous vs scheduled] |
| Silver transformation | [Tier] | [Sizing approach] | [Batch vs streaming] |
| Gold consumption | [Tier] | [Sizing approach] | [Query pattern] |

Cost-shaping levers (qualitative — concrete numbers belong in operational
runbooks, not the architecture):

- Spot / preemptible instances for retryable workloads.
- Auto-termination of idle clusters.
- Partition pruning and clustering for scan reduction.
- Materialised vs on-demand aggregates.

### Storage Strategy

| Layer | Format | Partitioning | Clustering / Optimisation |
|-------|--------|--------------|---------------------------|
| Bronze | [Delta / Iceberg / …] | [By date / source] | [Compaction policy] |
| Silver | [Format] | [By key / date] | [Z-order / cluster keys] |
| Gold | [Format] | [By query predicate] | [Materialised views / cache] |

### Platform Features in Use

| Feature | Use Case | Configuration Note |
|---------|----------|--------------------|
| [Auto Loader / equivalent] | [Bronze ingestion] | [Trigger mode, schema mode] |
| [Streaming Tables / equivalent] | [Bronze → Silver] | [Trigger / latency target] |
| [Pipeline orchestrator] | [End-to-end refresh] | [Schedule / dependency] |
| [Governance catalog] | [Access + lineage] | [Cross-team sharing posture] |

For non-Databricks platforms, see
[`docs/resources/databricks-platform-substitution.md`](../../../../../docs/resources/databricks-platform-substitution.md)
for the platform-equivalent terms.

## Decisions and Tradeoffs

### Key Architecture Decisions

| Decision | Choice | Rationale | Alternative Considered | Consequence |
|----------|--------|-----------|------------------------|-------------|
| [Medallion layering] | [Choice] | [Why] | [Alternative] | [Tradeoff] |
| [Streaming vs batch per layer] | [Choice] | [Why] | [Alternative] | [Tradeoff] |
| [Compute tier per workload] | [Choice] | [Why] | [Alternative] | [Tradeoff] |

### Performance vs Cost Tradeoffs

- [Real-time vs near-real-time ingestion — freshness gain vs sustained
  compute cost]
- [Materialised vs on-demand Gold aggregates — query latency vs storage]
- [Spot vs on-demand compute — cost savings vs interruption risk]

### Known Risks and Mitigations

| Risk | Mitigation |
|------|------------|
| [Source rate limit causes backlog] | [Backoff + queue buffering + lag alert] |
| [PII exposure in Bronze] | [Masked views + audit logs] |
| [Schema drift from source] | [Schema registry + manual approval gate] |

---

## Review Checklist

- [ ] **Scope** clearly states which data flows are in / out of bounds.
- [ ] **Medallion topology** names Bronze / Silver / Gold purposes and
  transformation rules.
- [ ] **Data flow diagrams** show how data moves through layers and to
  consumers.
- [ ] **Processing semantics** explicitly state streaming vs batch per layer
  with latency targets tied to [[prd]] (kind: data).
- [ ] **Pipeline-level quality contracts** name which contracts each layer
  boundary enforces; detailed `EXPECT` clauses are deferred to
  [[data-quality-expectations]].
- [ ] **Failure handling** specifies what happens when a contract fails
  (alert, reject, quarantine, rollback).
- [ ] **Access control** model covers identity, row-level, column-level,
  and sensitive-data masking at the policy level.
- [ ] **Platform design** names catalog organisation, compute tiering, and
  storage strategy without committing to hardcoded cost numbers.
- [ ] **Decisions and tradeoffs** document key choices with rationale and
  alternatives considered.
- [ ] **Cross-layer contracts** are defined (reconciliation, cardinality,
  no orphans).
- [ ] **SLA per layer** is documented (freshness, latency, availability)
  and traces to [[prd]] (kind: data).
- [ ] No `[TBD]`, `[TODO]`, or `[NEEDS CLARIFICATION]` markers remain.
- [ ] Entity-level details (logical schema, indexes, migrations, store
  selection) are deferred to [[data-design]].
- [ ] For non-Databricks platforms, terms map via
  `docs/resources/databricks-platform-substitution.md`.
diff --git a/docs/website/content/artifact-types/design/data-design.md b/docs/website/content/artifact-types/design/data-design.md index a713cb01..3e4d48e8 100644 --- a/docs/website/content/artifact-types/design/data-design.md +++ b/docs/website/content/artifact-types/design/data-design.md @@ -123,6 +123,6 @@ ddx: InformsTechnical Design
Test Plan HELIX documentsdocs/helix/02-design/data-design.md Generation prompt
Show the full generation prompt
# Data Design Generation Prompt
Document the data model and access patterns needed to support the design.

## Reference Anchors

Use this local resource summary as grounding:

- `docs/resources/fowler-evolutionary-database-design.md` grounds schema
  evolution, versioned migrations, data movement, and rollback expectations.

## Focus
- Name the main entities, stores, and key fields.
- Make relationships, lifecycle, and integrity constraints explicit.
- Capture the main access patterns and their performance or consistency needs.
- Note privacy, classification, retention, and protection consequences where they
  materially shape the design.
- Define migration and rollback expectations for schema or storage changes.
- Avoid drifting into implementation-specific query or ORM code.

## Role Boundary

Data Design is not the full architecture or implementation plan. It explains
the data model, storage responsibilities, access patterns, integrity/security
constraints, and migration consequences that technical designs must honor.

## Completion Criteria
- The model is understandable to another engineer without reading code.
- Key data decisions and constraints are explicit.
- Access patterns and migration strategy are concrete enough to guide
  implementation and tests.
-Template
Show the template structure
---
ddx:
  id: data-design
---

# Data Design

## Data Summary

- Scope: [What feature, subsystem, or workflow this data design supports]
- Storage systems: [Database, queue, cache, object store]
- Main concerns: [Consistency, scale, retention, privacy, migration]

## Entities and Stores

| Entity or Store | Purpose | Key Fields | Volume / Growth | Notes |
|-----------------|---------|------------|-----------------|-------|
| [Name] | [What it represents] | [Important fields] | [Expected scale] | [Business rules or constraints] |

## Relationships

| From | To | Type | Cardinality | On Delete |
|------|----|------|-------------|-----------|
| [Entity1] | [Entity2] | [1:N, N:M] | [Required/Optional] | [CASCADE/RESTRICT/SET NULL] |

## Access Patterns and Constraints

| Access Pattern | Frequency | Performance Need | Supporting Index or Cache |
|----------------|-----------|------------------|---------------------------|
| [Read or write path] | [Rate] | [Latency or throughput target] | [Index, partition, cache] |

## Validation and Security

| Field or Data Type | Rules / Classification | Protection or Error Handling |
|--------------------|------------------------|------------------------------|
| [Field] | [Constraints or classification] | [Masking, encryption, validation, retention] |

## Migration Strategy

- Tooling: [Migration framework]
- Approach: [Schema rollout and rollback strategy]
- Backfill or cleanup: [If needed]
+Template
Show the template structure
---
ddx:
  id: data-design
---

# Data Design

Entity-level data model for the feature or subsystem: logical entities,
stores, relationships, access patterns, integrity/security constraints, and
migration strategy. Platform-level concerns (medallion topology, processing
framework, governance model, pipeline-level quality contracts) live in
[[data-architecture]].

## Data Summary

- Scope: [What feature, subsystem, or workflow this data design supports]
- Storage systems: [Database, queue, cache, object store — names only; the
  platform-level rationale lives in [[data-architecture]]]
- Main concerns: [Consistency, scale, retention, privacy, migration]

## Entities and Stores

| Entity or Store | Purpose | Key Fields | Volume / Growth | Notes |
|-----------------|---------|------------|-----------------|-------|
| [Name] | [What it represents] | [Important fields] | [Expected scale] | [Business rules or constraints] |

## Relationships

| From | To | Type | Cardinality | On Delete |
|------|----|------|-------------|-----------|
| [Entity1] | [Entity2] | [1:N, N:M] | [Required/Optional] | [CASCADE/RESTRICT/SET NULL] |

## Access Patterns and Constraints

| Access Pattern | Frequency | Performance Need | Supporting Index or Cache |
|----------------|-----------|------------------|---------------------------|
| [Read or write path] | [Rate] | [Latency or throughput target] | [Index, partition, cache] |

## Validation and Security

Field-level rules. Pipeline-level masking and access policy live in
[[data-architecture]] (Governance and Access Control).

| Field or Data Type | Rules / Classification | Protection or Error Handling |
|--------------------|------------------------|------------------------------|
| [Field] | [Constraints or classification] | [Masking, encryption, validation, retention] |

## Migration Strategy

- Tooling: [Migration framework]
- Approach: [Schema rollout and rollback strategy]
- Backfill or cleanup: [If needed]

## Cross-References

- [[data-architecture]] — platform/pipeline shape, medallion topology,
  processing framework, governance model, and pipeline-level quality
  contracts.
- [[data-quality-expectations]] — executable field-level and freshness
  contracts that this model must satisfy.
diff --git a/docs/website/content/artifact-types/design/proof-of-concept.md b/docs/website/content/artifact-types/design/proof-of-concept.md index 15ca8afd..3c851000 100644 --- a/docs/website/content/artifact-types/design/proof-of-concept.md +++ b/docs/website/content/artifact-types/design/proof-of-concept.md @@ -205,6 +205,6 @@ column mapping scope. EnablesNone InformsSolution Design
Architecture
Contract
Implementation Plan
ADR Generation prompt
Show the full generation prompt
# Proof of Concept Prompt
Use the PoC to validate the smallest risky assumption that matters.

## Focus
- State the objective and success criteria clearly.
- Keep the approach small and the findings evidence-based.
- End with a decision or recommendation.
- Preserve enough implementation and test evidence for another engineer to reproduce the result.

## Role Boundary

Proof of Concept is not a tech spike, feature spec, or production design. It
records a small working implementation that proves or disproves a risky
technical approach. Tech Spike records investigation; PoC records working
evidence.

## Completion Criteria
- The hypothesis is tested.
- Results are easy to interpret.
- The next step is obvious.
- The production implications and remaining gaps are explicit.
-Template
Show the template structure
---
ddx:
  id: POC-XXX
---

# Proof of Concept: {{poc_title}}

**PoC ID**: {{poc_id}} | **Lead**: {{poc_lead}} | **Time Budget**: {{time_budget}} | **Status**: In Progress | Completed

## Objective

**Primary Question**: [What technical concept needs validation?]

**Success Criteria**:
- **Functional**: [Working implementation demonstrates X]
- **Performance**: [Baseline metric]
- **Integration**: [Systems involved]

**In Scope**: [Core functionality and key integrations]
**Out of Scope**: [Production hardening and full feature set]

## Approach

**Architecture Pattern**: [Approach to demonstrate]

**Key Technologies**:
- **Primary**: [Core stack]
- **Integration**: [External systems/APIs]

## Implementation

### Architecture Overview
```
[Architecture diagram or ASCII representation]
```

### Core Components
#### [Component Name]
- **Purpose**: [What it does]
- **Implementation**: [Technology and approach]

### Integration Points
| Integration | Type | Status | Notes |
|-------------|------|--------|--------|
| [System] | [API/DB/Queue] | [Working/Partial/Failed] | [Details] |

## Results

### Test Scenarios
| Scenario | Result | Status |
|----------|--------|--------|
| [Core workflow] | [What happened] | Pass/Fail/Partial |
| [Integration] | [What happened] | Pass/Fail/Partial |
| [Performance] | [What happened] | Pass/Fail/Partial |

### Findings
- **FINDING 1**: [Key discovery]
- **Evidence**: [Concrete proof]
- **Implications**: [What this means]

### Risks
| Risk | Prob | Impact | Mitigation |
|------|------|--------|------------|
| [Risk] | H/M/L | H/M/L | [Strategy] |

## Analysis

**Overall Assessment**: VIABLE | VIABLE WITH CONDITIONS | NOT VIABLE

**Rationale**: [Evidence-based assessment]

## Recommendations
1. [Action] -- [Rationale] -- [Timeline]

### Follow-up
- [ ] Design updates, additional PoCs, ADRs to create
+Template
Show the template structure
---
ddx:
  id: POC-XXX
---

# Proof of Concept: {{poc_title}}

**PoC ID**: {{poc_id}} | **Lead**: {{poc_lead}} | **Time Budget**: {{time_budget}} | **Status**: In Progress | Completed

## Objective

**Primary Question**: [What technical concept needs validation?]

**Success Criteria**:
- **Functional**: [Working implementation demonstrates X]
- **Performance**: [Baseline metric]
- **Integration**: [Systems involved]

**In Scope**: [Core functionality and key integrations]
**Out of Scope**: [Production hardening and full feature set]

## Approach

**Architecture Pattern**: [Approach to demonstrate]

**Key Technologies**:
- **Primary**: [Core stack]
- **Integration**: [External systems/APIs]

## Implementation

### Architecture Overview
```
[Architecture diagram or ASCII representation]
```

### Core Components
#### [Component Name]
- **Purpose**: [What it does]
- **Implementation**: [Technology and approach]

### Integration Points
| Integration | Type | Status | Notes |
|-------------|------|--------|--------|
| [System] | [API/DB/Queue] | [Working/Partial/Failed] | [Details] |

## Results

### Test Scenarios
| Scenario | Result | Status |
|----------|--------|--------|
| [Core workflow] | [What happened] | Pass/Fail/Partial |
| [Integration] | [What happened] | Pass/Fail/Partial |
| [Performance] | [What happened] | Pass/Fail/Partial |

### Findings
- **FINDING 1**: [Key discovery]
- **Evidence**: [Concrete proof]
- **Implications**: [What this means]

### Risks
| Risk | Prob | Impact | Mitigation |
|------|------|--------|------------|
| [Risk] | H/M/L | H/M/L | [Strategy] |

## Analysis

**Overall Assessment**: VIABLE | VIABLE WITH CONDITIONS | NOT VIABLE

**Rationale**: [Evidence-based assessment]

## Recommendations
1. [Action] -- [Rationale] -- [Timeline]

### Follow-up
- [ ] Design updates, additional PoCs, ADRs to create

## Artifacts

Preserved under `proofs-of-concept/{{poc_id}}/`:
- **Code**: [Working implementation files]
- **Data**: [Test data and fixtures]
- **Evidence**: [Logs, screenshots, benchmark output]
diff --git a/docs/website/content/artifact-types/design/solution-design.md b/docs/website/content/artifact-types/design/solution-design.md index 90c7a750..4cb8673b 100644 --- a/docs/website/content/artifact-types/design/solution-design.md +++ b/docs/website/content/artifact-types/design/solution-design.md @@ -16,8 +16,9 @@ model, component decomposition, interface usage, and requirement-to-design traceability. It applies Architecture, ADRs, Contracts, and Concerns to one feature or -cross-component capability. It does not redefine the system architecture. It -does not plan a single story's code changes; that belongs in Technical Design. +cross-component capability. For what belongs at this level versus Architecture +and Technical Design, see the zoom-stack matrix in +`workflows/activities/02-design/README.md`. ## Example @@ -252,7 +253,7 @@ Architecture and ADR-001. EnablesNone InformsTechnical Design
Test Plan HELIX documentsdocs/helix/02-design/solution-designs/SD-002-first-class-principles.md -Generation prompt
Show the full generation prompt
# Solution Design Generation Prompt

Create a solution design that maps requirements to a concrete approach.

## Purpose

Solution Design is the **feature-level design artifact**. Its unique job is to
translate a Feature Specification into a selected technical approach, domain
model, component decomposition, interface usage, and requirement-to-design
traceability.

It applies Architecture, ADRs, Contracts, and Concerns to one feature or
cross-component capability. It does not redefine the system architecture. It
does not plan a single story's code changes; that belongs in Technical Design.

## Reference Anchors

Use these local resource summaries as grounding:

- `docs/resources/arc42-solution-strategy.md` grounds feature-level approach,
  decomposition, interfaces, and concise design rationale.
- `docs/resources/c4-model.md` grounds component and interaction views that
  support feature-level decomposition.

## Focus
- Create a feature-level artifact named `docs/helix/02-design/solution-designs/SD-XXX-[name].md`.
- Show the main options and why the chosen one wins.
- Keep the domain model, decomposition, and tradeoffs concise.
- Cover cross-component system behavior and feature-level structure.
- Do not collapse into story-level implementation details; those belong in a
  technical design (`TD-XXX-*`).
- Preserve only the decisions needed by build and test.
- If the feature requires changing Architecture or an accepted ADR, stop and
  update that governing artifact first.

## Boundary Test

| If you are writing... | Put it in... |
|---|---|
| Product/feature behavior and acceptance criteria | Feature Specification |
| System-wide structure, deployment, and quality attributes | Architecture |
| One architecture-significant decision | ADR |
| Exact API/file/event/CLI surface | Contract |
| Feature-level approach and decomposition | Solution Design |
| Story-level code plan and file/module changes | Technical Design |

## Completion Criteria
- Requirements are mapped.
- Tradeoffs are explicit.
- The chosen approach is clear.
- The output is clearly feature-level and disambiguated from a technical
  design.
- Every P0 requirement has a corresponding design element and test strategy.
+Generation prompt
Show the full generation prompt
# Solution Design Generation Prompt

Create a solution design that maps requirements to a concrete approach.

## Purpose

Solution Design is the **feature-level design artifact**. Its unique job is to
translate a Feature Specification into a selected technical approach, domain
model, component decomposition, interface usage, and requirement-to-design
traceability.

It applies Architecture, ADRs, Contracts, and Concerns to one feature or
cross-component capability. For what belongs at this level versus Architecture
and Technical Design, see the zoom-stack matrix in
`workflows/activities/02-design/README.md`.

## Reference Anchors

Use these local resource summaries as grounding:

- `docs/resources/arc42-solution-strategy.md` grounds feature-level approach,
  decomposition, interfaces, and concise design rationale.
- `docs/resources/c4-model.md` grounds component and interaction views that
  support feature-level decomposition.

## Focus
- Create a feature-level artifact named `docs/helix/02-design/solution-designs/SD-XXX-[name].md`.
- Show the main options and why the chosen one wins.
- Keep the domain model, decomposition, and tradeoffs concise.
- Cover cross-component system behavior and feature-level structure.
- Preserve only the decisions needed by build and test.
- Stay within the feature scope defined in the zoom-stack matrix
  (`workflows/activities/02-design/README.md`); update the governing artifact
  first if the feature forces a change at a higher level.

## Boundary Test

See the zoom-stack matrix in `workflows/activities/02-design/README.md` for
which decisions belong at the system, feature, and story levels.

## Completion Criteria
- Requirements are mapped.
- Tradeoffs are explicit.
- The chosen approach is clear.
- The output is clearly feature-level and disambiguated from a technical
  design.
- Every P0 requirement has a corresponding design element and test strategy.
Template
Show the template structure
---
ddx:
  id: SD-XXX
  review:
    self_hash: ea6f092342409cc3f74e945b3ae421392eb4787113b828331c0fdfab359bf86d
    deps:
      FEAT-XXX: a685da86c4c18a509196cb163f264af507cc966f804db574070e108a555bdf02
    reviewed_at: "2026-05-15T04:11:24Z"
---

# Solution Design

**Feature**: [[FEAT-XXX]] | **Artifact**: `docs/helix/02-design/solution-designs/SD-XXX-[name].md`

## Scope

- Feature-level design artifact
- Use for cross-component behavior, main alternatives, domain model, and
  decomposition
- Do not use for one-story implementation details; those belong in `TD-XXX`
- Governing artifacts: [Architecture, ADRs, Contracts, Concerns]

## Requirements Mapping

### Functional Requirements

| Requirement | Technical Capability | Component | Priority |
|------------|---------------------|-----------|----------|
| [Business requirement] | [Technical implementation] | [Component] | P0/P1/P2 |

### NFR Impact on Architecture

| NFR | Requirement | Architectural Impact | Design Decision |
|-----|------------|---------------------|-----------------|
| Performance | [Metric] | [What this requires] | [How achieved] |
| Security | | | |
| Scalability | | | |

## Solution Approaches

### Approach 1: [Name]
**Description**: [Overview]
**Pros**: [Advantages]
**Cons**: [Disadvantages]
**Evaluation**: [Selected/Rejected: why]

### Approach 2: [Name]
[Same structure]

**Selected Approach**: [Which and why]

**Architecture/ADR impact**: [No change, or name required Architecture/ADR update]

## Domain Model

```mermaid
erDiagram
    %% [Define entities with attributes and relationships]
```

### Business Rules
1. [Rule]: [Description and implementation impact]

## System Decomposition

### Component: [Name]
- **Purpose**: [What it does]
- **Responsibilities**: [List]
- **Requirements Addressed**: [Which requirements]
- **Interfaces**: [How it communicates]
- **Owned by TDs**: [Story-level work that will be designed later]

### Component Interactions
```mermaid
graph TD
    %% [Show component relationships]
```

## Technology Rationale

Only include feature-specific technology choices here. System-wide choices
belong in Architecture or ADRs.

| Layer | Choice | Why | Alternatives Rejected |
|-------|--------|-----|----------------------|
| Language | [Choice] | [Reason] | [Others] |
| Framework | [Choice] | [Reason] | [Others] |
| Database | [Choice] | [Reason] | [Others] |
| Infrastructure | [Choice] | [Reason] | [Others] |

## Traceability

| Requirement ID | Component | Design Element | Test Strategy |
|---------------|-----------|----------------|---------------|
| FR-001 | [Component] | [How addressed] | [How tested] |

### Gaps
- [ ] [Requirement not fully addressed]: [Mitigation]

## Concern Alignment

If the project has active concerns (`docs/helix/01-frame/concerns.md`), confirm
this design is consistent with them:

- **Concerns used**: [Which active concerns does this design rely on?]
- **Constraints honored**: [Any concern constraints that shaped this design?]
- **ADRs referenced**: [Concern-related ADRs that govern design choices here]
- **Departures**: [Any design choices that depart from concern practices? If so,
  an ADR should justify the departure.]

## Constraints & Assumptions

- **Constraints**: [Technical constraints and their design impact]
- **Assumptions**: [What we assume, risk if wrong]
- **Dependencies**: [External systems, libraries]

## Risks

| Risk | Prob | Impact | Mitigation |
|------|------|--------|------------|
| [Risk] | H/M/L | H/M/L | [Strategy] |

## Review Checklist

Use this checklist when reviewing a solution design:

- [ ] Requirements mapping covers all P0 functional requirements from the governing spec
- [ ] NFR impact section shows how the architecture satisfies each non-functional requirement
- [ ] At least two solution approaches were evaluated with concrete pros/cons
- [ ] Selected approach rationale explains why alternatives were rejected
- [ ] Domain model captures all entities and their relationships
- [ ] Business rules are specific enough to implement
- [ ] System decomposition assigns every requirement to at least one component
- [ ] Component interfaces are defined — not just names, but how they communicate
- [ ] Technology rationale explains why each choice was made, not just what was chosen
- [ ] Traceability table maps every requirement to a component and test strategy
- [ ] Gaps section lists any requirements not fully addressed with mitigation plans
- [ ] Concern alignment verifies consistency with active project concerns
- [ ] Design is consistent with governing feature spec and PRD
diff --git a/docs/website/content/artifact-types/design/technical-design.md b/docs/website/content/artifact-types/design/technical-design.md index ae3d9f75..8beaf076 100644 --- a/docs/website/content/artifact-types/design/technical-design.md +++ b/docs/website/content/artifact-types/design/technical-design.md @@ -15,10 +15,10 @@ unique job is to make one user story buildable by naming the concrete component changes, files, interfaces, data model changes, security implications, tests, rollback path, and implementation sequence. -It inherits Architecture and Solution Design. It must not redesign the feature -or system. If the story cannot be implemented without changing the parent -Solution Design, ADR, Contract, or Architecture, update that governing artifact -first. +It inherits Architecture and Solution Design. For what belongs at this level +versus those higher levels, see the zoom-stack matrix in +`workflows/activities/02-design/README.md`; if the story forces a change at a +higher level, update that governing artifact first. ## Example @@ -55,18 +55,6 @@ ddx: - Does not implement column mapping, row validation, import confirmation, or match generation. -## Acceptance Criteria - -1. **Given** Maya is viewing Acme Dental, **When** she uploads one valid bank - CSV and one valid invoice CSV, **Then** DepositMatch creates one draft import - session for Acme Dental and opens mapping review. -2. **Given** Maya is viewing Acme Dental, **When** she uploads a PDF instead of - a CSV for either required file, **Then** DepositMatch rejects the file before - parsing and keeps the import session in draft. -3. **Given** Maya has uploaded both required CSV files, **When** the files are - accepted, **Then** the import session records the client, file names, upload - time, and source type for each file. - ## Technical Approach **Strategy**: Implement the API-001 upload contract in the Fastify API and add a @@ -196,16 +184,18 @@ CREATE TABLE import_files ( ## Testing -- [ ] **Unit**: `importUploadService` rejects missing files, non-CSV files, and - inaccessible clients. -- [ ] **Integration**: API route returns API-001 success shape and stores draft - session/file metadata in one transaction. -- [ ] **API**: Contract tests for 201, 400 `missing-import-file`, 415 - `unsupported-import-file-type`, and 503 `import-storage-unavailable`. +Each governing-story AC-ID is realized below (ADR-009 — AC text lives in [[US-001]], not here): + +- [ ] **Unit** (US-001-AC1, US-001-AC2): `importUploadService` rejects missing + files, non-CSV files, and inaccessible clients. +- [ ] **Integration** (US-001-AC1, US-001-AC3): API route returns API-001 + success shape and stores draft session/file metadata in one transaction. +- [ ] **API** (US-001-AC2): Contract tests for 201, 400 `missing-import-file`, + 415 `unsupported-import-file-type`, and 503 `import-storage-unavailable`. - [ ] **Security**: Verify raw CSV row values are absent from logs for failed and successful uploads. -- [ ] **UI**: Upload component routes to `next.href` after successful upload and - renders problem-details errors. +- [ ] **UI** (US-001-AC1): Upload component routes to `next.href` after + successful upload and renders problem-details errors. ## Migration & Rollback @@ -259,7 +249,7 @@ in development/test environments. EnablesNone InformsTest Plan
Implementation Plan HELIX documentsdocs/helix/02-design/technical-designs/TD-012-artifact-types-navigation.md -Generation prompt
Show the full generation prompt
# Technical Design for User Story Prompt

Create a concise technical design for one user story.

## Purpose

Technical Design is the **story-level implementation design artifact**. Its
unique job is to make one user story buildable by naming the concrete component
changes, files, interfaces, data model changes, security implications, tests,
rollback path, and implementation sequence.

It inherits Architecture and Solution Design. It must not redesign the feature
or system. If the story cannot be implemented without changing the parent
Solution Design, ADR, Contract, or Architecture, update that governing artifact
first.

## Reference Anchors

Use these local resource summaries as grounding:

- `docs/resources/google-small-cls.md` grounds bounded implementation slices
  with related tests and rollback.
- `docs/resources/cucumber-executable-specifications.md` grounds mapping
  acceptance criteria to observable tests.

## Active Concerns

For each concern selected in `docs/helix/01-frame/concerns.md`, apply its declared
`## Artifact Impact` (from `workflows/concerns/<name>/concern.md`) to THIS technical design — realize the
TD-level obligations it names (domain-driven-design -> aggregates/value-objects/repositories; architecture-style -> layering + dependency direction; cqrs -> command/query split). A selected concern whose Artifact Impact names TD
but leaves no trace here is drift (reconcile-alignment Concern->Artifact Realization check).

## Focus
- Create a story-level artifact named `docs/helix/02-design/technical-designs/TD-XXX-[name].md`.
- Map each acceptance criterion to component changes, interfaces, data, security, and tests.
- Stay on the vertical slice for the story.
- Assume the broader architecture is already set by the parent solution design.
- Do not expand into a feature-wide or system-wide design; that belongs in a
  solution design (`SD-XXX-*`).
- Keep implementation sequence and rollout or migration notes only when they affect execution.

## Boundary Test

| If you are writing... | Put it in... |
|---|---|
| Feature-wide approach or decomposition | Solution Design |
| One architectural decision | ADR |
| Exact external interface contract | Contract |
| One story's implementation shape, files, tests, and rollback | Technical Design |
| Test fixtures and detailed test cases | Story Test Plan |
| Work queue slicing and execution status | Implementation Plan or runtime work item |

## Completion Criteria
- The story is implementable.
- Key interfaces, changes, and test coverage are explicit.
- The design stays compact.
- The output is clearly story-level and disambiguated from a solution design.
- The implementation sequence can be turned into one or more small,
  reviewable changes without losing test coverage.
-Template
Show the template structure
---
ddx:
  id: TD-XXX
  review:
    self_hash: 081ac39c2360ed0034e2a9bc05b5932fbd2baa2930b605c2ab947bf4548a2015
    deps:
      FEAT-XXX: a685da86c4c18a509196cb163f264af507cc966f804db574070e108a555bdf02
      SD-XXX: ea6f092342409cc3f74e945b3ae421392eb4787113b828331c0fdfab359bf86d
      US-XXX: 48b416257cf7acd8b225b785edcb09a125fed67521af9c8f115ec7dc2fbf23a3
    reviewed_at: "2026-05-15T04:11:24Z"
---

# Technical Design: TD-XXX-[story-name]

**User Story**: [[US-XXX]] | **Feature**: [[FEAT-XXX]] | **Solution Design**: [[SD-XXX]]

## Scope

- Story-level design artifact
- Use for one vertical slice or one bounded implementation story
- Must inherit the broader approach from the parent solution design
- Do not redefine cross-component architecture here; that belongs in `SD-XXX`
- Governing artifacts: [User Story, Solution Design, Contracts, Concerns]

## Acceptance Criteria

1. **Given** [precondition], **When** [action], **Then** [expected outcome]
2. **Given** [precondition], **When** [action], **Then** [expected outcome]

## Technical Approach

**Strategy**: [Brief description]

**Key Decisions**:
- [Decision]: [Rationale]

**Trade-offs**:
- [What we gain vs. lose]

## Component Changes

### Modified: [Component Name]
- **Current State**: [What exists]
- **Changes**: [What changes]
- **Files**: `[path]`

### New: [Component Name]
- **Purpose**: [Why needed]
- **Interfaces**: Input: [receives] / Output: [produces]
- **Files**: `[path]`

## API/Interface Design

```yaml
endpoint: /api/v1/[resource]
method: POST
request:
  type: object
  properties:
    field1: string
response:
  type: object
  properties:
    id: string
    status: string
```

## Data Model Changes

```sql
-- New tables or schema modifications
CREATE TABLE [table_name] (
    id UUID PRIMARY KEY,
    [columns]
);
```

## Integration Points

| From | To | Method | Data |
|------|-----|--------|------|
| [Source] | [Target] | [REST/Event/Direct] | [What data] |

### External Dependencies
- **[Service]**: [Usage] | Fallback: [If unavailable]

## Security

- **Authentication**: [Required auth level]
- **Authorization**: [Required permissions]
- **Data Protection**: [Encryption/masking]
- **Threats**: [Specific threats and mitigations]

## Performance

- **Expected Load**: [Requests/sec, data volume]
- **Response Target**: [Milliseconds]
- **Optimizations**: [Caching, indexing, etc.]

## Testing

- [ ] **Unit**: [What to test]
- [ ] **Integration**: [What integrations to test]
- [ ] **API**: [Endpoints to test]
- [ ] **Security**: [Security scenarios]

## Migration & Rollback

- **Backward Compatibility**: [Strategy]
- **Data Migration**: [Required migrations]
- **Feature Toggle**: [Enable/disable mechanism]
- **Rollback**: [Steps to reverse]

## Implementation Sequence

1. [What to build first] -- Files: `[paths]` -- Tests: `[paths]`
2. [What to build next]
3. [Integration and verification]

**Prerequisites**: [Dependencies that must be complete first]

## Risks

| Risk | Prob | Impact | Mitigation |
|------|------|--------|------------|
| [Risk] | H/M/L | H/M/L | [Strategy] |

## Review Checklist

Use this checklist when reviewing a technical design:

- [ ] Acceptance criteria use Given/When/Then format and are verifiable
- [ ] Technical approach inherits from the parent solution design — no contradictions
- [ ] Key decisions have documented rationale
- [ ] Trade-offs are explicit — what we gain and what we lose
- [ ] Component changes clearly describe current state vs. changes
- [ ] API/interface design includes request and response schemas
- [ ] Data model changes include migration SQL
- [ ] Integration points specify fallback behavior for external dependencies
- [ ] Security section addresses authentication, authorization, and data protection
- [ ] Performance targets are numeric with specific metrics
- [ ] Testing section covers unit, integration, API, and security scenarios
- [ ] Migration and rollback strategy is documented
- [ ] Implementation sequence is ordered with file paths and test paths
- [ ] Design is consistent with governing solution design and feature spec
+Generation prompt
Show the full generation prompt
# Technical Design for User Story Prompt

Create a concise technical design for one user story.

## Purpose

Technical Design is the **story-level implementation design artifact**. Its
unique job is to make one user story buildable by naming the concrete component
changes, files, interfaces, data model changes, security implications, tests,
rollback path, and implementation sequence.

It inherits Architecture and Solution Design. For what belongs at this level
versus those higher levels, see the zoom-stack matrix in
`workflows/activities/02-design/README.md`; if the story forces a change at a
higher level, update that governing artifact first.

## Reference Anchors

Use these local resource summaries as grounding:

- `docs/resources/google-small-cls.md` grounds bounded implementation slices
  with related tests and rollback.
- `docs/resources/cucumber-executable-specifications.md` grounds mapping
  acceptance criteria to observable tests.

## Active Concerns

For each concern selected in `docs/helix/01-frame/concerns.md`, apply its declared
`## Artifact Impact` (from `workflows/concerns/<name>/concern.md`) to THIS technical design — realize the
TD-level obligations it names (domain-driven-design -> aggregates/value-objects/repositories; architecture-style -> layering + dependency direction; cqrs -> command/query split). A selected concern whose Artifact Impact names TD
but leaves no trace here is drift (reconcile-alignment Concern->Artifact Realization check).

## Focus
- Create a story-level artifact named `docs/helix/02-design/technical-designs/TD-XXX-[name].md`.
- Realize each governing-story AC-ID (US-{n}-AC{m}) through component changes, interfaces, data, security, and tests; reference AC-IDs, do not restate AC text (ADR-009 — AC ownership lives in user-stories).
- Stay on the vertical slice for the story, within the story scope defined in
  the zoom-stack matrix (`workflows/activities/02-design/README.md`).
- Keep implementation sequence and rollout or migration notes only when they affect execution.

## Boundary Test

See the zoom-stack matrix in `workflows/activities/02-design/README.md` for
which decisions belong at the system, feature, and story levels.

## Completion Criteria
- The story is implementable.
- Key interfaces, changes, and test coverage are explicit.
- The design stays compact.
- The output is clearly story-level and disambiguated from a solution design.
- The implementation sequence can be turned into one or more small,
  reviewable changes without losing test coverage.
+Template
Show the template structure
---
ddx:
  id: TD-XXX
  review:
    self_hash: 081ac39c2360ed0034e2a9bc05b5932fbd2baa2930b605c2ab947bf4548a2015
    deps:
      FEAT-XXX: a685da86c4c18a509196cb163f264af507cc966f804db574070e108a555bdf02
      SD-XXX: ea6f092342409cc3f74e945b3ae421392eb4787113b828331c0fdfab359bf86d
      US-XXX: 48b416257cf7acd8b225b785edcb09a125fed67521af9c8f115ec7dc2fbf23a3
    reviewed_at: "2026-05-15T04:11:24Z"
---

# Technical Design: TD-XXX-[story-name]

**User Story**: [[US-XXX]] | **Feature**: [[FEAT-XXX]] | **Solution Design**: [[SD-XXX]]

## Scope

- Story-level design artifact
- Use for one vertical slice or one bounded implementation story
- Must inherit the broader approach from the parent solution design
- Do not redefine cross-component architecture here; that belongs in `SD-XXX`
- Governing artifacts: [User Story, Solution Design, Contracts, Concerns]

## Technical Approach

**Strategy**: [Brief description]

**Key Decisions**:
- [Decision]: [Rationale]

**Trade-offs**:
- [What we gain vs. lose]

## Component Changes

### Modified: [Component Name]
- **Current State**: [What exists]
- **Changes**: [What changes]
- **Files**: `[path]`

### New: [Component Name]
- **Purpose**: [Why needed]
- **Interfaces**: Input: [receives] / Output: [produces]
- **Files**: `[path]`

## API/Interface Design

```yaml
endpoint: /api/v1/[resource]
method: POST
request:
  type: object
  properties:
    field1: string
response:
  type: object
  properties:
    id: string
    status: string
```

## Data Model Changes

```sql
-- New tables or schema modifications
CREATE TABLE [table_name] (
    id UUID PRIMARY KEY,
    [columns]
);
```

## Integration Points

| From | To | Method | Data |
|------|-----|--------|------|
| [Source] | [Target] | [REST/Event/Direct] | [What data] |

### External Dependencies
- **[Service]**: [Usage] | Fallback: [If unavailable]

## Security

- **Authentication**: [Required auth level]
- **Authorization**: [Required permissions]
- **Data Protection**: [Encryption/masking]
- **Threats**: [Specific threats and mitigations]

## Performance

- **Expected Load**: [Requests/sec, data volume]
- **Response Target**: [Milliseconds]
- **Optimizations**: [Caching, indexing, etc.]

## Testing

- [ ] **Unit**: [What to test]
- [ ] **Integration**: [What integrations to test]
- [ ] **API**: [Endpoints to test]
- [ ] **Security**: [Security scenarios]

## Migration & Rollback

- **Backward Compatibility**: [Strategy]
- **Data Migration**: [Required migrations]
- **Feature Toggle**: [Enable/disable mechanism]
- **Rollback**: [Steps to reverse]

## Implementation Sequence

1. [What to build first] -- Files: `[paths]` -- Tests: `[paths]`
2. [What to build next]
3. [Integration and verification]

**Prerequisites**: [Dependencies that must be complete first]

## Risks

| Risk | Prob | Impact | Mitigation |
|------|------|--------|------------|
| [Risk] | H/M/L | H/M/L | [Strategy] |

## Review Checklist

Use this checklist when reviewing a technical design:

- [ ] Each governing-story AC-ID (US-{n}-AC{m}) is realized by the technical changes (AC text is not restated here — ADR-009)
- [ ] Technical approach inherits from the parent solution design — no contradictions
- [ ] Key decisions have documented rationale
- [ ] Trade-offs are explicit — what we gain and what we lose
- [ ] Component changes clearly describe current state vs. changes
- [ ] API/interface design includes request and response schemas
- [ ] Data model changes include migration SQL
- [ ] Integration points specify fallback behavior for external dependencies
- [ ] Security section addresses authentication, authorization, and data protection
- [ ] Performance targets are numeric with specific metrics
- [ ] Testing section covers unit, integration, API, and security scenarios
- [ ] Migration and rollback strategy is documented
- [ ] Implementation sequence is ordered with file paths and test paths
- [ ] Design is consistent with governing solution design and feature spec
diff --git a/docs/website/content/artifact-types/frame/_index.md b/docs/website/content/artifact-types/frame/_index.md index 547c9d98..c16df3a5 100644 --- a/docs/website/content/artifact-types/frame/_index.md +++ b/docs/website/content/artifact-types/frame/_index.md @@ -13,7 +13,7 @@ Define what the system should do, for whom, and how success will be measured. {{< card link="prd/" title="Product Requirements Document" subtitle="Product-scope authority document that translates vision into prioritized, measurable requirements, success metrics, non-goals, assumptions, and acceptance sket…" >}} {{< card link="principles/" title="Project Principles" subtitle="Cross-cutting design principles that guide judgment calls across all HELIX activities. They are not requirements, concerns, ADRs, workflow rules, or process en…" >}} {{< card link="concerns/" title="Project Concerns" subtitle="Declares the project's active cross-cutting concerns: technology stacks, quality attributes, data rules, security posture, UX conventions, operating practices,…" >}} - {{< card link="feature-specification/" title="Feature Specification" subtitle="Feature-level authority document that translates PRD requirements into precise behavior, boundaries, functional areas, non-functional expectations, acceptance…" >}} + {{< card link="feature-specification/" title="Feature Specification" subtitle="Feature-level authority document that translates PRD requirements into precise behavior, boundaries, functional areas, non-functional expectations, edge cases,…" >}} {{< card link="user-stories/" title="User Stories" subtitle="Stable design artifacts defining vertical user journeys through feature behavior. Each story traces one persona from trigger to outcome, references parent feat…" >}} {{< /cards >}} @@ -31,5 +31,4 @@ Define what the system should do, for whom, and how success will be measured. {{< card link="stakeholder-map/" title="Stakeholder Map" subtitle="Stakeholder and decision-rights map for roles, interests, influence, engagement, RACI ownership, communication cadence, and escalation paths." >}} {{< card link="threat-model/" title="Threat Model" subtitle="Structured analysis of assets, data flows, trust boundaries, STRIDE threats, risk, mitigations, owners, and verification hooks." >}} {{< card link="validation-checklist/" title="Validation Checklist" subtitle="Frame exit gate that verifies completeness, consistency, traceability, evidence, stakeholder review, and readiness to enter Design." >}} - {{< card link="data-prd/" title="Data Product Requirements Document" subtitle="Data-scoped authority document that translates data vision into prioritized, measurable requirements, data quality standards, data consumer expectations, and D…" >}} {{< /cards >}} diff --git a/docs/website/content/artifact-types/frame/compliance-requirements.md b/docs/website/content/artifact-types/frame/compliance-requirements.md index d0972be3..e53c3100 100644 --- a/docs/website/content/artifact-types/frame/compliance-requirements.md +++ b/docs/website/content/artifact-types/frame/compliance-requirements.md @@ -185,7 +185,7 @@ and confirm legal applicability before production rollout. EnablesNone InformsSolution Design Referenced byTest Plan -Generation prompt
Show the full generation prompt
# Compliance Requirements Analysis Prompt
Document the compliance obligations for this project in the local template.

## Reference Anchors

Use these local resource summaries as grounding:

- `docs/resources/ftc-safeguards-rule.md` grounds financial customer
  information security obligations and applicability caveats.
- `docs/resources/nist-privacy-framework.md` grounds privacy risk management,
  data processing, controls, and validation evidence.

## Focus
- Identify only the regulations and standards that actually apply to this system.
- Mark uncertain applicability explicitly and route it to counsel or compliance review.
- Map each obligation to its source, affected scope, concrete controls, owners, evidence, and timing.
- Keep the result concise and implementation-relevant.
- Do not invent legal conclusions. State assumptions and review gaps.

## Role Boundary

Compliance Requirements is not Security Requirements or the Threat Model. It
defines external obligations and the controls/evidence needed to satisfy them.
Security Requirements turns those obligations into security behavior; Threat
Model analyzes abuse paths and mitigations.

## Completion Criteria
- Applicable, not-applicable, and uncertain obligations are identified.
- Controls, evidence, owners, and deadlines are explicit.
- No generic filler is added.
+Generation prompt
Show the full generation prompt
# Compliance Requirements Analysis Prompt
Map the external regulations and standards that apply to this project to the
controls and evidence that satisfy them.

## Traceability chain

This artifact's place in the security triangle: **regulations -> controls**.
- See `security-requirements` for the testable acceptance criteria that
  exercise those controls.
- See `threat-model` for the abuse paths those controls mitigate and the
  STRIDE owners.

## Reference Anchors

Use these local resource summaries as grounding:

- `docs/resources/ftc-safeguards-rule.md` grounds financial customer
  information security obligations and applicability caveats.
- `docs/resources/nist-privacy-framework.md` grounds privacy risk management,
  data processing, controls, and validation evidence.

## Focus
- Identify only the regulations and standards that actually apply to this system.
- Mark uncertain applicability explicitly and route it to counsel or compliance review.
- Map each obligation to its source, affected scope, the control(s) that satisfy
  it, owners, evidence, and timing.
- Keep the result concise and implementation-relevant.
- Do not invent legal conclusions. State assumptions and review gaps.
- Do not enumerate threats, attacker behavior, or STRIDE categories. Cross-
  reference `threat-model` for those.
- Do not author testable acceptance criteria for controls. Cross-reference
  `security-requirements` for that.

## Completion Criteria
- Applicable, not-applicable, and uncertain obligations are identified.
- Each applicable obligation names the control(s), evidence, owner, and deadline.
- No generic filler is added.
Template
Show the template structure
---
ddx:
  id: compliance-requirements
---

# Compliance Requirements

**Project**: [Project Name]
**Compliance Risk Level**: [Critical/High/Medium/Low]
**Date**: [Creation Date]

## Executive Summary

**Applicable Regulations**: [List of regulations that apply]
**Compliance Scope**: [What aspects of the system must comply]
**Key Requirements**: [Top 3-5 highest-impact requirements]

## Applicable Regulations

For each applicable or uncertain regulation, document:

### [Regulation Name]
- **Jurisdiction**: [Where it applies]
- **Applicability**: Applies / Does not apply / Needs counsel review
- **Assumption**: [Why it applies or does not apply to this project]
- **Key Requirements**: [Summary of main requirements]
- **Penalties**: [Potential fines/sanctions]
- **Timeline**: [Any specific deadlines]

### Industry Standards

#### [Standard Name]
- **Scope**: [What aspects are covered]
- **Certification Required**: [Yes/No]
- **Key Controls**: [Summary of main controls]

## Compliance Requirements Matrix

For each regulation, create a requirements table:

| Requirement | Reference | Description | Implementation Control | Evidence | Owner | Status |
|-------------|-----------|-------------|------------------------|----------|-------|--------|
| [Requirement] | [Article/Section] | [What must be done] | [How] | [Audit artifact] | [Who] | [Status] |

## Data Classification and Handling

| Data Type | Classification | Regulations | Handling Requirements |
|-----------|----------------|-------------|----------------------|
| [Type] | [Sensitivity level] | [Applicable regs] | [Required controls] |

## Applicability Gaps

| Question | Why It Matters | Owner | Due Date |
|----------|----------------|-------|----------|
| [Open legal/compliance question] | [Decision affected] | [Owner] | [Date] |

### Data Retention

| Data Type | Retention Period | Legal Basis | Disposal Method |
|-----------|------------------|-------------|-----------------|
| [Type] | [Duration] | [Why] | [How] |

## Privacy Requirements

### Data Subject Rights (if applicable)

| Right | Implementation | Response Time |
|-------|----------------|---------------|
| [Right] | [How implemented] | [SLA] |

### Privacy Impact Assessment

For each data processing activity:
- **Data Types**: [What data]
- **Purpose**: [Why processed]
- **Legal Basis**: [Lawful basis]
- **Risk Level**: [High/Medium/Low]

## Incident Response and Reporting

### Breach Notification Requirements

| Regulation | Authority Notification | Individual Notification | Timeline |
|------------|----------------------|------------------------|----------|
| [Reg] | [Requirements] | [Requirements] | [Deadline] |

## Compliance Risk Assessment

| Risk | Impact | Likelihood | Risk Level | Mitigation |
|------|--------|------------|------------|------------|
| [Risk] | [Impact] | [Likelihood] | [Level] | [Action] |

## Implementation Plan

- [ ] Regulatory applicability analysis and gap assessment
- [ ] Data protection, access controls, audit logging
- [ ] Privacy mechanisms and data subject rights
- [ ] Vendor data processing agreements
- [ ] Compliance testing and audits
diff --git a/docs/website/content/artifact-types/frame/data-prd.md b/docs/website/content/artifact-types/frame/data-prd.md deleted file mode 100644 index d8686afa..00000000 --- a/docs/website/content/artifact-types/frame/data-prd.md +++ /dev/null @@ -1,208 +0,0 @@ ---- -title: "Data Product Requirements Document" -linkTitle: "Data Prd" -slug: data-prd -activity: "Frame" -artifactRole: "supporting" -weight: 90 -generated: true ---- - -## Purpose - -The Data PRD is the **data-product-scope authority for what data to build and why**. -Its unique job is to translate business intent into data-centric requirements: -data sources, consumer personas, quality contracts, technical constraints (catalog, -schema, medallion layer), and measurable success metrics. It sits between the -general Product Vision and Data Architecture. Every data pipeline design choice -and quality expectation should trace back to a Data PRD requirement. - -## Example - -
-Show a worked example of this artifact - -``````markdown ---- -ddx: - id: example.data-prd.customer-360 ---- - -# Data Product Requirements Document: Customer-360 Analytics - -## Summary - -Customer-360 is a unified analytics dataset that reconciles Salesforce customer -accounts with Stripe payment history and customer lifecycle events. It powers -sales forecasting, churn analysis, and account health scoring. The first release -ingests 12 months of historical data and supports daily incremental updates for -ongoing revenue and subscription trends. Success means sales analysts can query a -single Gold table to answer "total ARR per customer," "months since last payment," -and "invoice aging" without joining across multiple systems. - -## Problem and Goals - -### Problem - -Sales and finance teams use Salesforce for CRM and Stripe for payment processing, -but data is siloed. Analysts manually pull exports, reconcile customer IDs across -systems, and build fragile spreadsheets to answer cash-flow and churn questions. -Each reconciliation takes 3-4 hours and risks stale data by the time insights are -shared. - -### Goals - -1. Unify Salesforce accounts and Stripe customers under a single customer identity. -2. Preserve payment lineage: which invoice, subscription, or charge event drove - each revenue transaction. -3. Deliver Gold tables that require no post-query transformation for common - business questions. - -### Success Metrics - -| Metric | Target | Measurement Method | -|--------|--------|--------------------| -| Query response time | Under 5 seconds for customer-year aggregations | Databricks query execution time | -| Data freshness | Daily updates complete before 7am UTC | Pipeline job logs and SLA monitoring | -| Customer ID reconciliation accuracy | 98% of Salesforce accounts matched to Stripe customers | Manual audit sample of matched pairs | -| Analyst adoption | 10+ scheduled reports consuming Gold tables | Databricks workspace audit logs | - -### Non-Goals - -- Streaming ingestion (batch-daily is sufficient for v1). -- Custom churn-scoring algorithms (featurized data enables external ML). -- Integration with Marketo, HubSpot, or third-party data warehouses. -- Supporting sub-account hierarchies within Stripe. - -## Data Consumers - -| Role | Team | Key Questions | Tables | -|------|------|---------------|--------| -| Sales Analyst | Revenue | ARR by account, churn risk score, invoice aging | dim_customer, fct_monthly_revenue | -| Finance Manager | Revenue Operations | Subscription changes, failed payment count, billing exceptions | fct_subscription_event, fct_payment_transaction | -| Data Engineer | Analytics | Lineage validation, late-arriving facts, reconciliation record counts | all layer tables, metadata.processing_log | - -## Data Sources - -| Source System | Entity | Volume (12-month history) | Frequency | SLA | -|---------------|--------|---------------------------|-----------|-----| -| Salesforce | Account records (customer name, owner, industry) | 5,000 accounts | Daily export (API) | 6-hour lag acceptable | -| Salesforce | Opportunity records (pipeline, closed deals) | 50,000 opps | Daily export (API) | 6-hour lag acceptable | -| Stripe | Customer objects (email, metadata tags) | 4,500 customers | Real-time webhook (v1 batch daily) | 24-hour batch acceptable for v1 | -| Stripe | Subscription objects (plan, status, start/end dates) | 6,200 subscriptions | Real-time webhook (v1 batch daily) | 24-hour batch acceptable for v1 | -| Stripe | Invoice records (amount, status, line items) | 45,000 invoices | Real-time webhook (v1 batch daily) | 24-hour batch acceptable for v1 | -| Stripe | Charge records (card, amount, outcome) | 180,000 charges | Real-time webhook (v1 batch daily) | 24-hour batch acceptable for v1 | - -## Data Quality Requirements - -### Coverage - -| Expectation | Threshold | Owner | Severity | -|-------------|-----------|-------|----------| -| Salesforce account export completeness | ≥ 95% of prior day's account count | Data Eng | P0 (block load) | -| Stripe customer export completeness | ≥ 95% of prior day's distinct customers | Data Eng | P0 (block load) | -| Customer ID reconciliation rate | ≥ 98% of matched Salesforce-Stripe pairs | Data Eng | P1 (alert, allow backfill) | -| Invoice line-item accuracy | 100% of invoices ≥ $0; amount matches sum of line items | Data Eng | P0 (block load) | - -### Freshness - -| Dataset | Expected latency | Acceptable Delay | Monitoring | -|---------|------------------|------------------|------------| -| Bronze Salesforce (raw export) | Within 6 hours of API call | 8 hours | Query SLA dashboard | -| Bronze Stripe (raw export) | Within 24 hours of event (batch v1) | 26 hours | Query SLA dashboard | -| Silver (deduplicated, reconciled) | Within 8 hours of Bronze completion | 10 hours | Pipeline job history | -| Gold (aggregated facts) | Before 7am UTC daily | 8am UTC | Scheduled report execution | - -### Metadata Requirements - -- **Lineage**: Every Gold row must record the source Salesforce and Stripe record IDs, - import batch ID, and last-modified timestamp. -- **Data Quality Flags**: Silver layer must include reconciliation match confidence, - late-arriving fact flags, and anomaly detection scores. -- **Processing Metadata**: All tables must include `_loaded_at`, `_processed_by_job_id`, - and `_source_system`. - -## Databricks-Specific Technical Context - -### Workspace and Catalog - -- **Catalog**: main (Unity Catalog enabled) -- **Schema Naming**: customer_360_{bronze,silver,gold} -- **Medallion Layers**: Separate schemas for isolation and access control - -### Compute - -- **Orchestration**: Databricks Workflows with daily 10pm UTC trigger -- **Compute**: Job cluster, 2 workers, 8 DBU/hour estimate -- **Language**: Python with PySpark for orchestration, SQL for transformations - -### Access Control - -- **Bronze**: Data Engineer read/write; Analyst no access (sensitive raw PII) -- **Silver**: Data Engineer read/write; Analyst read-only (internal team only) -- **Gold**: Data Engineer read/write; Analyst read; Managers read (published via BI) - -### Compliance - -- **PII Handling**: Customer email and phone hashed in Silver+ layers; raw email in Bronze only -- **Retention**: Bronze/Silver retained 90 days; Gold retained 3 years -- **Masking**: Stripe card tokens never stored; card brand only in Silver - -## Constraints and Assumptions - -### Constraints - -- **Data Volume**: 12-month history = ~250GB uncompressed; daily increments ~1GB -- **Cost**: ≤ $500 USD/month DBU spend for batch jobs + queries -- **Latency**: Daily batch only for v1; no streaming/real-time SLAs - -### Assumptions - -- Salesforce and Stripe customer data stabilizes within 24 hours of creation -- No single customer appears as multiple accounts in Salesforce (will validate in reconciliation) -- Email is a sufficient match key for Salesforce-Stripe reconciliation (vs. manual linking) - -### Dependencies - -- Salesforce API credentials and rate-limit quota -- Stripe API credentials and historical export access -- Databricks Unity Catalog workspace with ≥ 2 worker nodes available - -## Risks - -| Risk | Probability | Impact | Mitigation | -|------|-------------|--------|------------| -| Email format inconsistency blocks Stripe-Salesforce joins | High | High | Implement fuzzy email matching (lowercase, domain normalization) in Silver | -| Stripe subscription deletion removes invoice lineage | Medium | High | Copy foreign keys to Bronze before join; archive deleted subscriptions in reference table | -| Gold table query cost exceeds budget | Medium | Medium | Add partition pruning by month; set query-timeout alarms; publish consumption dashboard | - -## Open Questions - -- [ ] Does Salesforce contain sub-accounts or partner-account hierarchies? Will this affect the 1:1 customer assumption? -- [ ] Are there known Stripe test customers in the export that should be filtered in Bronze? -- [ ] Which BI tool will query Gold tables (Tableau, Looker, Sigma)? Will it handle incremental refresh? - -## Success Criteria - -Customer-360 is successful when: -1. Analysts execute fewer than 5 manual export-and-reconcile cycles per quarter. -2. Sales team's monthly revenue forecast uses only Gold table queries (no spreadsheets). -3. Churn alerting dashboard refreshes successfully every morning by 7am UTC for 30 consecutive days. -`````` - -
- -## Reference - - - - - - - - - - - - -
ActivityFrame — Define what the system should do, for whom, and how success will be measured.
Default locationdocs/helix/01-frame/data-prd.md
RequiresNone
EnablesNone
InformsData Architecture
Feature Specification
User Stories
Referenced byData Architecture
Technical Design
Generation prompt
Show the full generation prompt
# Data Product Requirements (Data PRD) Generation Prompt

Create a Data PRD that frames the data product problem, scope, quality requirements,
and success criteria clearly enough that downstream data architecture, quality
expectations, and implementation work can trace back to it.

## Purpose

The Data PRD is the **data-product-scope authority for what data to build and why**.
Its unique job is to translate business intent into data-centric requirements:
data sources, consumer personas, quality contracts, technical constraints (catalog,
schema, medallion layer), and measurable success metrics. It sits between the
general Product Vision and Data Architecture. Every data pipeline design choice
and quality expectation should trace back to a Data PRD requirement.

## Reference Anchors

Use these local resource summaries as grounding:

- `docs/resources/databricks-unity-catalog.md` grounds data governance through
  unified catalog hierarchies (metastore → catalog → schema → volume/table).
- `docs/resources/databricks-lakehouse-medallion-architecture.md` grounds
  medallion topology (Bronze/Silver/Gold) and layer responsibilities in a
  Lakehouse.
- `docs/resources/databricks-sdp.md` grounds Databricks Semantic Data Platform
  governance, lineage, and quality contracts through `EXPECT ... ON VIOLATION ...`
  clauses and SDP-aware pipeline patterns.

## Focus

- Name the data sources (internal and external), ingestion cadence, and expected
  volume/velocity.
- Define data consumers and their use cases: who consumes the data, what decisions
  they make, and what quality they depend on.
- List data quality requirements as testable contracts: completeness, accuracy,
  freshness, schema consistency, referential integrity.
- Specify the Databricks technical context: target catalog/schema, medallion
  layer (Bronze/Silver/Gold), pipeline type (Auto Loader, Streaming Tables, SQL
  pipeline), and estimated DBU budget.
- Frame success metrics for the data product itself: SLA compliance (freshness,
  availability), consumer satisfaction, cost per GB, defect escape rate.

## Role Boundary

Data PRD is not a general product PRD, data model, pipeline design, or quality
implementation. It specifies *what* data requirements are, not *how* to implement
them.

**Databricks Platform Substitution:** If you are adopting this on another data
platform, substitute as follows:

| Databricks Concept | Snowflake Equivalent | BigQuery Equivalent | On-Prem / Other |
|---|---|---|---|
| Unity Catalog (UC) hierarchy | Database + Schema + Table | Project + Dataset + Table | Database + Schema |
| Medallion architecture (Bronze/Silver/Gold) | Same pattern applies universally | Same pattern applies universally | Same pattern applies universally |
| Auto Loader, Streaming Tables | Snowpipe, Stream-triggered tasks | Dataflow, BigQuery Streaming Inserts | Apache Spark Structured Streaming, Airflow |
| SDP `EXPECT ... ON VIOLATION ...` | Data Quality checks in Snowflake | BigQuery Data Quality API | dbt tests, Great Expectations assertions |
| DBU budget estimation | Credit consumption | On-demand or flat-rate pricing | Compute resource allocation |

## Completion Criteria

- Data sources table names each external/internal source, ingestion pattern, and
  freshness requirement.
- Data consumers table includes consumer role, use case, and quality SLA.
- Quality requirements are specific constraints (not aspirational); each one is
  testable in Data Quality Expectations.
- Technical context names the target catalog, schema, medallion layer, and
  pipeline type.
- Success metrics have numeric targets and measurement methods (e.g., "SLA
  compliance > 95% measured by on-time delivery vs. promised refresh cadence").
- Requirements trace upward to the Product Vision or general PRD and downward
  to Data Architecture and Data Quality Expectations.
Template
Show the template structure
---
ddx:
  id: data-prd
---

# Data Product Requirements Document

## Executive Summary

[This section works as a standalone 1-pager. Include: what data product we are building, who uses it, what business problem it solves, the data solution approach, and the top 2-3 success metrics. Write this last — it should be a distillation of the full PRD, not an introduction. Someone reading only this section should understand the data product well enough to decide whether to read the rest.]

## Problem and Goals

### Problem Statement

[What is broken or missing in the current data landscape? Who is affected? Be specific: not "users struggle with reporting" but "sales analysts spend 4 hours per week reconciling pipeline outputs with source systems because current freshness is 24 hours and source data changes hourly."]

### Business Goals

1. [Primary goal — what changes for the organization or data consumers]
2. [Secondary goal — secondary business impact or risk reduction]

### Goals and Objectives

| Goal | Objective | Success Criteria |
|------|-----------|------------------|
| [Goal] | [How we achieve it via data] | [Measurable outcome] |

### Non-Goals

[What we are explicitly not trying to achieve. Each non-goal should exclude something a reasonable person might assume is in scope.]

## Data Consumers

### Primary Consumer: [Name/Role]

**Team**: [Data Engineering, Analytics, Product, Finance, etc.]
**Use Case**: [What do they do with this data? How does it change their workflow?]
**Frequency**: [Real-time, daily, weekly, ad-hoc?]
**Key Tables/Feeds**: [Which outputs matter most to this consumer?]

### Secondary Consumer: [Name/Role]

[Repeat above structure for each secondary consumer.]

### Data Consumer Requirements Table

| Consumer | Use Case | Freshness SLA | Latency Tolerance | Key Dimensions | Access Level |
|----------|----------|---------------|-------------------|----------------|--------------|
| [Team] | [What they do] | [e.g., hourly] | [max delay] | [customer_id, product_id, ...] | [Row-level, Column-level, or Full] |

## Data Sources

### Source System Inventory

| Source System | Schema / Table | Owner | Update Frequency | Quality Baseline | Notes |
|---------------|----------------|-------|------------------|------------------|-------|
| [e.g., Salesforce] | [e.g., Accounts, Opportunities] | [Team] | [hourly, daily, on-demand] | [% completeness, freshness] | [Data model version, API limits, retry policy] |

## Requirements Overview

### P0 (Must Have)

[Critical requirements blocking data product delivery or violating SLA.]

- Requirement: [e.g., "Ingest Salesforce data within 1 hour of transaction close"]
- Requirement: [e.g., "Support daily rollup of customer spend by product category"]

### P1 (Should Have)

[Important for end-user value or operational stability.]

- Requirement: [e.g., "Detect and alert on data quality anomalies within 15 minutes"]
- Requirement: [e.g., "Enable drill-down to daily granularity without latency penalty"]

### P2 (Nice to Have)

[Nice-to-have enhancements deferred if time-constrained.]

- Requirement: [e.g., "Publish historical snapshots for year-over-year comparison"]

## Data Quality Requirements

### Quality Dimensions and Thresholds

| Dimension | P0 Threshold | P1 Threshold | Measurement Method | Enforcement |
|-----------|--------------|--------------|-------------------|-------------|
| Completeness | [e.g., ≥99%] | [e.g., ≥95%] | [Count NULLs / total rows] | [Alert if falls below P0] |
| Timeliness | [e.g., ≤1 hour lag] | [e.g., ≤4 hour lag] | [MAX(ingestion_time) - MAX(source_time)] | [Reject data if exceeds P0] |
| Accuracy | [e.g., ≥98% match to source] | [e.g., ≥95% match] | [Row-count reconciliation + sample audit] | [Manual review + auto-reject if P0 fails] |
| Uniqueness | [e.g., PK has no duplicates] | [as P0] | [COUNT(*) = COUNT(DISTINCT PK)] | [Fail ingestion] |

### Data Quality Expectations

[Reference the [[data-quality-expectations]] document for detailed EXPECT clauses per medallion layer. This section summarizes which quality dimensions are critical.]

## Databricks-Specific Technical Context

### Catalog and Schema Strategy

- **Target Catalog**: [e.g., `prod`, `analytics`, or domain-specific catalog]
- **Target Schema**: [e.g., `customer_360`, `payment_events`]
- **Medallion Layers**: Bronze (raw), Silver (validated), Gold (business)
- **Access Control Model**: [UC policies, Row-level security, column masking]

### Medallion Layer Strategy

- **Bronze**: [Raw data ingestion point; what is ingested, what constraints apply, how often]
- **Silver**: [Validated and deduplicated; transformation rules, quality gates]
- **Gold**: [Business-ready tables; aggregations, dimensions, fact tables for consumers]

### Databricks Features and Compute Strategy

| Feature | Decision | Rationale |
|---------|----------|-----------|
| Ingestion Pattern | [Auto Loader, Streaming Tables, batch] | [Why this choice?] |
| Processing Model | [Streaming, Batch, Incremental] | [Freshness SLA and cost tradeoff] |
| Compute Tier | [All-purpose, Jobs, Serverless] | [Workload characteristics, cost model] |
| Storage Format | [Delta, Parquet, CSV] | [Durability, query performance needs] |
| DBU Budget (Monthly) | [Estimated spend] | [Based on row volume, freshness, complexity] |

### Governance and Compliance

- **Data Classification**: [Public, Internal, Sensitive, PII]
- **Retention Policy**: [e.g., Bronze: 7 days, Silver: 90 days, Gold: 2 years]
- **Audit Trail**: [Who accessed what, when, why]
- **Lineage Tracking**: [Table-to-table dependencies for impact analysis]

## Success Metrics

| Metric | Target | Baseline | Measurement Method | Cadence |
|--------|--------|----------|-------------------|---------|
| [Throughput] | [e.g., 1M rows/day] | [Current: 100K rows/day] | [COUNT(*) from production table] | Daily |
| [Latency] | [e.g., ≤1 hour end-to-end] | [Current: 4 hours] | [MAX(ingestion_timestamp) - MAX(source_timestamp)] | Hourly |
| [Quality Score] | [e.g., ≥98%] | [Current: 85%] | [Automated quality checks pass rate] | Daily |
| [Cost per GB] | [e.g., $0.05/GB/month] | [Current: $0.12/GB/month] | [DBU spend / data volume] | Monthly |

## Risks and Mitigation

| Risk | Likelihood | Impact | Mitigation Strategy |
|------|------------|--------|-------------------|
| [e.g., Source API rate limits] | [High] | [Data ingestion delays] | [Implement backoff + queue; alert on throttling] |
| [e.g., Schema drift in source] | [Medium] | [Pipeline failure or silent corruption] | [Schema registry; alerts on new columns; manual review] |
| [e.g., Cost overrun from compute] | [Medium] | [Budget overrun] | [Set DBU limits; implement cost monitoring] |

---

## Review Checklist

Use this checklist during review to validate that the data PRD is complete and ready for design:

- [ ] **Executive Summary** is self-contained and clearly states the data product purpose
- [ ] **Problem Statement** is specific: includes failure mode, frequency, and affected stakeholders (not generic)
- [ ] **Data Consumers table** names actual teams/roles with concrete use cases (not abstract personas)
- [ ] **Data Sources** table is complete: every source system has an entry with owner and update frequency
- [ ] **Requirements** are prioritized (P0/P1/P2) and traceable to data consumer use cases
- [ ] **Quality Dimensions** have numeric thresholds (not vague targets like "high quality")
- [ ] **Databricks Context** specifies catalog, schema, medallion layer strategy, and compute approach
- [ ] **Success Metrics** are quantified: throughput (rows/day), latency (max age), quality score (%), cost ($/GB)
- [ ] **Risks** are specific with likelihood/impact assessment and mitigation tactics
- [ ] No `[TBD]`, `[TODO]`, or `[NEEDS CLARIFICATION]` markers remain
- [ ] Naming and terminology align with Databricks Unity Catalog and SDP conventions
- [ ] Every P0 requirement has at least one data quality expectation (link to [[data-quality-expectations]])
diff --git a/docs/website/content/artifact-types/frame/feasibility-study.md b/docs/website/content/artifact-types/frame/feasibility-study.md index a9605414..bc32ab48 100644 --- a/docs/website/content/artifact-types/frame/feasibility-study.md +++ b/docs/website/content/artifact-types/frame/feasibility-study.md @@ -176,6 +176,6 @@ only with pilot recruiting, compliance review, and explicit success metrics. EnablesNone InformsPRD
Principles
Risk Register
Stakeholder Map
Feature Registry Generation prompt
Show the full generation prompt
# Feasibility Study Generation Prompt
Assess whether the project is feasible and what it would take to proceed.

## Reference Anchors

Use these local resource summaries as grounding:

- `docs/resources/doj-feasibility-study.md` grounds pre-commitment
  feasibility, alternatives, decision criteria, and recommendation.
- `docs/resources/eib-project-feasibility.md` grounds option analysis,
  cost-benefit, organizational, compliance, and risk dimensions.

## Focus
- Separate technical, business, operational, and resource feasibility.
- Compare realistic alternatives, including delaying or doing nothing where useful.
- State the recommendation clearly.
- Capture the main risks, constraints, and open questions.
- Tie conclusions to evidence and confidence, not optimism.

## Role Boundary

Feasibility Study is not the Business Case, PRD, or Solution Design. It decides
whether the opportunity is viable enough to justify deeper framing or delivery
commitment. Business Case owns investment return; PRD owns required behavior;
Solution Design owns the chosen implementation approach.

## Completion Criteria
- The recommendation is unambiguous.
- Each feasibility dimension is summarized briefly.
- Assumptions and mitigations are explicit.
- The preferred alternative is justified against at least one rejected alternative.
-Template
Show the template structure
---
ddx:
  id: feasibility-study
---

# Feasibility Study: {{project_name}}

**Decision Deadline**: {{decision_deadline}}
**Status**: Draft

## Executive Summary

### Project Overview
[Brief description of the proposed project or solution]

### Recommendation
**Overall Assessment**: FEASIBLE | CONDITIONALLY FEASIBLE | NOT FEASIBLE
**Decision**: GO | CONDITIONAL GO | NO GO
**Rationale**: [2-3 sentences on the main factors]
**Confidence**: High | Medium | Low

## Feasibility Assessment

### Technical
- **Assessment**: FEASIBLE | HIGH RISK | NOT FEASIBLE
- **Key requirements**: [Brief list]
- **Main risks**: [Brief list]
- **Evidence**: [What supports the assessment]

### Business
- **Assessment**: FEASIBLE | HIGH RISK | NOT FEASIBLE
- **Market opportunity**: [Brief summary]
- **Value proposition**: [Brief summary]
- **Evidence**: [What supports the assessment]

### Operational
- **Assessment**: FEASIBLE | HIGH RISK | NOT FEASIBLE
- **Support and deployment needs**: [Brief summary]
- **Regulatory requirements**: [Brief summary]
- **Evidence**: [What supports the assessment]

### Resource
- **Assessment**: FEASIBLE | HIGH RISK | NOT FEASIBLE
- **Budget**: [Estimate]
- **Team and timeline**: [Estimate]
- **Evidence**: [What supports the assessment]

## Risks

| Risk | Probability | Impact | Mitigation |
|------|-------------|--------|------------|
| [Risk] | High/Med/Low | High/Med/Low | [Strategy] |

## Alternatives

### [Alternative Approach]
- **Pros**: [Brief]
- **Cons**: [Brief]
- **Feasibility**: [Brief]
- **Decision**: Carry forward | Reject

### Do Nothing / Delay
- **Pros**: [Brief]
- **Cons**: [Brief]
- **Feasibility**: [Brief]
- **Decision**: Carry forward | Reject

## Next Steps
1. [Action]
2. [Action]
+Template
Show the template structure
---
ddx:
  id: feasibility-study
---

# Feasibility Study: {{project_name}}

**Decision Deadline**: {{decision_deadline}}
**Status**: Draft

## Executive Summary

### Project Overview
[Brief description of the proposed project or solution]

### Recommendation
**Overall Assessment**: FEASIBLE | CONDITIONALLY FEASIBLE | NOT FEASIBLE
**Decision**: GO | CONDITIONAL GO | NO GO
**Rationale**: [2-3 sentences on the main factors]
**Confidence**: High | Medium | Low

## Feasibility Assessment

### Technical
- **Assessment**: FEASIBLE | HIGH RISK | NOT FEASIBLE
- **Key requirements**: [Brief list]
- **Main risks**: [Brief list]
- **Evidence**: [What supports the assessment]

### Business
- **Assessment**: FEASIBLE | HIGH RISK | NOT FEASIBLE
- **Market opportunity**: [Brief summary]
- **Value proposition**: [Brief summary]
- **Evidence**: [What supports the assessment]

### Operational
- **Assessment**: FEASIBLE | HIGH RISK | NOT FEASIBLE
- **Support and deployment needs**: [Brief summary]
- **Regulatory requirements**: [Brief summary]
- **Evidence**: [What supports the assessment]

### Resource
- **Assessment**: FEASIBLE | HIGH RISK | NOT FEASIBLE
- **Budget**: [Estimate]
- **Team and timeline**: [Estimate]
- **Evidence**: [What supports the assessment]

## Risks

| Risk | Probability | Impact | Mitigation |
|------|-------------|--------|------------|
| [Risk] | High/Med/Low | High/Med/Low | [Strategy] |

## Alternatives

### [Alternative Approach]
- **Pros**: [Brief]
- **Cons**: [Brief]
- **Feasibility**: [Brief]
- **Decision**: Carry forward | Reject

### Do Nothing / Delay
- **Pros**: [Brief]
- **Cons**: [Brief]
- **Feasibility**: [Brief]
- **Decision**: Carry forward | Reject

## Decision Framework

| Criterion | Status | Rationale |
|-----------|--------|-----------|
| Technical buildability | Pass / Risk / Fail | [Rationale] |
| Business value | Pass / Risk / Fail | [Rationale] |
| Operational supportability | Pass / Risk / Fail | [Rationale] |
| Compliance readiness | Pass / Risk / Fail | [Rationale] |
| Resource availability | Pass / Risk / Fail | [Rationale] |

## Next Steps
1. [Action]
2. [Action]
diff --git a/docs/website/content/artifact-types/frame/feature-registry.md b/docs/website/content/artifact-types/frame/feature-registry.md index aba27c5a..5ff79e54 100644 --- a/docs/website/content/artifact-types/frame/feature-registry.md +++ b/docs/website/content/artifact-types/frame/feature-registry.md @@ -113,7 +113,7 @@ ddx: RequiresNone EnablesNone Referenced byProject Dashboard
Release Notes
Progress Reports -Generation prompt
Show the full generation prompt
# Feature Registry Generation Prompt
Maintain the feature registry as the source of truth for IDs, status, dependencies, ownership, and traceability.

## Reference Anchors

Use these local resource summaries as grounding:

- `docs/resources/ibm-requirements-management.md` grounds requirements
  traceability, prioritization, validation, and change management.
- `docs/resources/atlassian-product-backlog.md` grounds visible prioritized
  work, dependency awareness, and refinement.

## Focus
- Assign new FEAT-XXX IDs sequentially.
- Keep status changes and dependencies explicit.
- Preserve traceability to stories, designs, contracts, tests, and code.
- Keep descriptions short; detail belongs in feature specs, stories, designs, and tests.

## Role Boundary

The Feature Registry is not the PRD, backlog, or tracker. It assigns durable
feature identity and preserves feature-level traceability. The PRD defines
requirements; Feature Specifications define behavior; runtime work items track execution.

## Completion Criteria
- Entries are brief and complete.
- IDs are unique and never reused.
- The registry stays easy to scan.
- Every active feature links to its governing artifact or clearly states the missing link.
+Generation prompt
Show the full generation prompt
# Feature Registry Generation Prompt
Maintain the feature registry as the source of truth for IDs, status, dependencies, ownership, and traceability.

## Reference Anchors

Use these local resource summaries as grounding:

- `docs/resources/ibm-requirements-management.md` grounds requirements
  traceability, prioritization, validation, and change management.
- `docs/resources/atlassian-product-backlog.md` grounds visible prioritized
  work, dependency awareness, and refinement.

## Focus
- Assign new FEAT-XXX IDs sequentially.
- Keep status changes and dependencies explicit.
- Preserve traceability to stories, designs, contracts, tests, and code.
- Keep descriptions short; detail belongs in feature specs, stories, designs, and tests.

## Role Boundary

The Feature Registry is not the PRD, backlog, or tracker. It assigns durable
feature identity and preserves feature-level traceability. The PRD defines
requirements; Feature Specifications define behavior; runtime work items track execution.

## Completion Criteria
- Entries are brief and complete.
- IDs are unique and never reused.
- The registry stays easy to scan.
- Every active feature links to its governing artifact or clearly states the missing link.

## Promotion from Parking Lot

Per [ADR-010](../../../../../docs/helix/02-design/adr/ADR-010-feature-registry-parking-lot-handoff.md),
`feature-registry` and `parking-lot` stay separate and the handoff is an
explicit recorded transition. When a parking-lot entry's revisit criteria are
met, promote it to the registry with the following procedure.

### Promotion Criteria

A parking-lot entry is eligible for promotion when all of these hold:

- **Revisit trigger fired**: the objective condition recorded on the entry has
  occurred (date reached, dependency landed, external signal observed).
- **Scope decided**: the item has been re-scoped into something a feature spec
  can be written against — not a vague idea kept warm.
- **Owner assigned**: a named owner accepts responsibility for the feature
  through at least the `specified` status.
- **Blocking ADRs resolved**: any ADR that the entry was waiting on has landed
  (accepted or rejected); items still pending ADRs stay parked.
- **Dependencies available**: prerequisite features listed on the entry are at
  a status that unblocks this one (typically `built` or later).

If any criterion fails, leave the entry parked and update the rationale or
revisit trigger to reflect what is still missing.

### Promotion Procedure

1. **Assign the next sequential FEAT-XXX**: never reuse an ID, including IDs
   from cancelled or deprecated features. Add the new row to `Active Features`
   with initial status `Draft` (or `Specified` if the spec is ready to land in
   the same change).
2. **Record the back-link to the parking-lot source**: in the new feature row's
   `Source` column, cite the parking-lot entry title (e.g.
   `parking-lot:<entry-title>`). This makes the parked-to-active transition
   auditable.
3. **Update the parking-lot entry**: mark the entry as promoted, record the
   assigned `FEAT-XXX`, and the promotion date. Do not delete the parking-lot
   entry — the historical record is part of the back-link.
4. **Seed traceability**: populate the new feature's `Trace Links` row with the
   feature spec, stories, designs, tests, and release placeholders. Empty cells
   are fine; missing cells are not.
5. **Carry over dependencies**: copy any `Dependencies` from the parking-lot
   entry into the registry's `Dependencies` table, expressed as FEAT-to-FEAT
   edges where the prerequisites have FEAT-XXX IDs.

The promotion is complete when the new `FEAT-XXX` row exists with a back-link,
the parking-lot entry records the promotion, and traceability rows are seeded.
Template
Show the template structure
---
ddx:
  id: feature-registry
---

# Feature Registry

**Status**: [Active | Archived]
**Last Updated**: [Date]

## Active Features

| ID | Name | Description | Status | Priority | Owner | Source | Updated |
|----|------|-------------|--------|----------|-------|--------|---------|
| FEAT-001 | [Name] | [Brief description] | [Status] | P0 | [Owner] | [PRD/spec/story] | [Date] |

## Status Definitions

- **Draft**: Requirements being gathered
- **Specified**: Feature spec complete (Frame done)
- **Designed**: Technical design complete (Design done)
- **In Test**: Tests being written
- **In Build**: Implementation in progress
- **Built**: Implementation complete
- **Deployed**: Released to production
- **Deprecated**: Scheduled for removal
- **Cancelled**: Will not be pursued

## Dependencies

| Feature | Depends On | Type | Notes |
|---------|------------|------|-------|
| FEAT-002 | FEAT-001 | Required | [Why] |

## Trace Links

| Feature | Spec | Stories | Designs | Tests | Release |
|---------|------|---------|---------|-------|---------|
| FEAT-001 | [Feature spec] | [Stories] | [Designs] | [Tests] | [Release] |

## Feature Categories

### [Category Name]
- FEAT-XXX: [Feature Name]

## ID Rules

1. Sequential numbering: FEAT-XXX (zero-padded 3 digits)
2. Never reuse IDs, even for cancelled features
3. Do not encode category or priority into the ID
4. Keep full behavior in Feature Specifications, not in this registry

## Deprecated/Cancelled

| ID | Name | Status | Reason | Date |
|----|------|--------|--------|------|
| FEAT-XXX | [Name] | [Cancelled/Deprecated] | [Why] | [Date] |
diff --git a/docs/website/content/artifact-types/frame/feature-specification.md b/docs/website/content/artifact-types/frame/feature-specification.md index a832545e..da2e05ee 100644 --- a/docs/website/content/artifact-types/frame/feature-specification.md +++ b/docs/website/content/artifact-types/frame/feature-specification.md @@ -12,8 +12,8 @@ generated: true A feature spec is the **feature-level authority for behavior and boundaries**. It translates PRD requirements into precise feature behavior, functional areas, -acceptance criteria, non-functional expectations, edge cases, and -feature-specific success measures. +non-functional expectations, edge cases, and feature-specific success measures. +Acceptance criteria belong to user stories (ADR-009) and are not defined here. It sits between the PRD (which defines product scope) and user stories (which define vertical slices through the feature). The feature spec owns feature @@ -27,15 +27,14 @@ how the behavior will be built. explains why the change is needed; it should not be the only organizing frame. - **Scope, not solution** — describe what the feature must do, not how to build it. Implementation details belong in design docs. -- **Behavior, not journey** — specify feature behavior and acceptance criteria. - Put end-to-end user flow narrative in user stories. +- **Behavior, not journey** — specify feature behavior and boundaries. + Put end-to-end user flow narrative and acceptance criteria in user stories. - **One feature, one capability** — a feature spec covers exactly one capability (≈ one PRD subsystem). If it covers two, split it; apply the Decomposition test below to decide. A functional *area* is a sub-part of one capability, not a second capability. - **Functional areas before requirements** — when a feature spans multiple surfaces, stages, or domain objects *within the one capability*, name those - areas before writing requirements and group requirements by area instead of _Additional guidance continues in the full prompt below._ @@ -53,7 +52,6 @@ committing. - [ ] Similar domain objects are separated before requirements are written - [ ] Functional requirements are grouped by area when a flat list would mix unrelated scopes - [ ] Every functional requirement is testable -- [ ] Acceptance criteria cover the highest-risk requirements with observable examples - [ ] Non-functional requirements have specific numeric targets - [ ] User stories are referenced by ID (not duplicated inline) - [ ] Dependencies name specific feature IDs and external systems @@ -267,7 +265,7 @@ evidence, exception records, and reconciliation exports. EnablesNone InformsUser Stories
Solution Design
Api Contracts
Test Plan HELIX documentsdocs/helix/01-frame/features/FEAT-013-runtime-install-coverage.md -Generation prompt
Show the full generation prompt
# Feature Specification Generation Prompt

Create a feature specification that is precise enough to support design,
user story creation, and test planning.

## Storage Location

Store at: `docs/helix/01-frame/features/FEAT-NNN-<name>.md`

## Purpose

A feature spec is the **feature-level authority for behavior and boundaries**.
It translates PRD requirements into precise feature behavior, functional areas,
acceptance criteria, non-functional expectations, edge cases, and
feature-specific success measures.

It sits between the PRD (which defines product scope) and user stories (which
define vertical slices through the feature). The feature spec owns feature
behavior. User stories own user journeys. Solution and technical designs own
how the behavior will be built.

## Reference Anchors

Use these local resource summaries as grounding:

- `docs/resources/ibm-requirements-management.md` grounds traceable,
  prioritized, verifiable requirements.
- `docs/resources/cucumber-executable-specifications.md` grounds concrete
  examples as readable acceptance specifications without prescribing
  implementation or tooling.

## Active Concerns

For each concern selected in `docs/helix/01-frame/concerns.md`, apply its declared
`## Artifact Impact` (from `workflows/concerns/<name>/concern.md`) to THIS feature spec — realize the
FEAT-level obligations it names (usage-metering -> which actions are billable; multi-tenancy -> tenant-scoped ACs). A selected concern whose Artifact Impact names FEAT
but leaves no trace here is drift (reconcile-alignment Concern->Artifact Realization check).

## Key Principles

- **Future state before current pain** — describe the desired user-visible
  outcome before optimizing around today's broken surface. The problem statement
  explains why the change is needed; it should not be the only organizing frame.
- **Scope, not solution** — describe what the feature must do, not how to
  build it. Implementation details belong in design docs.
- **Behavior, not journey** — specify feature behavior and acceptance criteria.
  Put end-to-end user flow narrative in user stories.
- **One feature, one capability** — a feature spec covers exactly one capability
  (≈ one PRD subsystem). If it covers two, split it; apply the Decomposition test
  below to decide. A functional *area* is a sub-part of one capability, not a
  second capability.
- **Functional areas before requirements** — when a feature spans multiple
  surfaces, stages, or domain objects *within the one capability*, name those
  areas before writing requirements and group requirements by area instead of
  producing one flat list. (Areas are subordinate parts of one capability — if an
  "area" would pass the Decomposition test as its own capability, it is a separate
  feature, not an area.)
- **Separate similar domain objects** — if readers might confuse two things,
  define them separately before requirements. For example, "Artifacts" are
  project-specific instances; "Artifact Types" are reusable methodology
  definitions.
- **Stories by reference** — list user story IDs, don't duplicate story
  content. Stories are separate files with their own lifecycle.
- **Testable requirements** — every functional requirement should be
  verifiable. If you can't describe how to test it, it's too vague.
- **Concrete acceptance examples** — add examples for important rules,
  permissions, errors, and edge cases. They should show observable behavior,
  not internal steps.
- **Leave unknowns explicit** — use Open Questions at the bottom rather than
  inventing detail you don't have.

## Boundary Test

| If you are writing... | Put it in... |
|---|---|
| Product goals, personas, launch priority, or product-level metrics | PRD |
| Feature behavior, boundaries, acceptance criteria, and edge cases | Feature Specification |
| A vertical user journey through one or more feature requirements | User Story |
| Component choices, data model, APIs, or implementation approach | Solution/Technical Design |
| Detailed test cases, fixtures, or automation strategy | Test Plan or Story Test Plan |
| Build sequencing and work slices | Implementation Plan |

## Decomposition — is it a FEAT or a functional area?

The brief decomposes into features at one granularity: **one feature per
capability**, anchored to the PRD's `### Subsystem:` groupings (~one subsystem →
one `FEAT-NNN`). Use these **layered tests** to place a candidate:

1. **Primary — ship / cut / metric.** A candidate is its own **feature** if all
   hold:
   - **Ship/cut:** it could be removed or deferred **without making another
     *named* capability incoherent** (it stands alone in the parking-lot).
   - **Metric:** it carries its own **feature-level product/user outcome** as a
     success metric — not a local counter (a button click or a row count is not a
     feature metric).
   If a candidate fails these — it cannot stand alone and has no outcome of its
   own — it is a **functional area** within a feature, not a feature.
2. **Tie-breaker — bounded context.** When ship/cut is genuinely ambiguous, split
   on bounded context / aggregate root: one feature per bounded context; areas are
   views/stages over the *same* aggregate.

**Anchor:** the PRD names the subsystems; each maps to ~one feature. A
multi-subsystem brief that produces a single mega-feature, or that produces zero
feature specs (PRD → stories directly), has skipped this tier — reconcile-alignment
flags both. A deliberately cross-subsystem feature (the workflow that spans them
*is* the feature) is allowed, but must say so explicitly in the template's
**Cross-Subsystem Rationale** field (the "Covered PRD Subsystem(s)" /
"Covered PRD Requirements" fields hold the subsystem names and FR IDs).
## Section-by-Section Guidance

### Overview
Connect this feature to a specific PRD requirement. "This feature implements
PRD P0-3" is better than "This feature improves the user experience."

### Ideal Future State
Describe the target state in user-visible terms. A good future state answers:

- What can the user understand, decide, or accomplish?
- What does the product surface make clear?
- How should the feature feel when it is working well?

For IA, documentation, onboarding, workflow, or product-surface features, this
section is mandatory. It should lead the spec toward the desired experience,
not merely away from the current failure mode.

### Problem Statement
Same standard as the PRD: describe the failure mode, not the absence of your
feature. Quantify where possible. Keep it subordinate to the future state; do
not let the spec become a list of current complaints.

### Functional Areas
Use this section whenever a feature has more than one surface, stage, or domain
object **within its one capability**. The area map should make clear what belongs
where before requirements are written. Areas are *subordinate parts* of the
feature — each fails the Decomposition test on its own (it cannot ship/cut
independently and has no feature-level outcome of its own).

Examples (areas *inside one capability*):

- CSV lead import → field mapping, validation, duplicate handling, confirmation
- Template editor → block palette, variable insertion, live preview, save/version
- Campaign scheduler → recipient selection, send-time rules, blackout handling

**Caution:** lists of *roles* ("Admin, Operator, Auditor"), *lifecycle stages*
("Intake, Planning, Execution, Review"), or *distinct domain objects* ("Leads,
Lists, Segments", "API, CLI, docs") are usually **separate features**, not areas
of one — each typically passes the Decomposition test as its own capability.
Apply the test before treating them as areas.

### Functional Requirements
Number each requirement for traceability. Group requirements by functional
area when the feature spans multiple areas. Use stable prefixes that make the
scope clear (`NAV-01`, `TYPE-01`, `ART-01`) or use plain `FR-01` for narrow
single-area features.

Each requirement should be independently testable. These are what the feature
must do — user stories describe how users interact with these capabilities.

If a requirement mentions two areas joined by "and", split it unless the
relationship between those areas is itself the requirement.

### Acceptance Criteria
Capture observable examples for the highest-risk or most important
requirements. Use concise Given/When/Then phrasing if it helps, but do not
require Cucumber tooling. Each example should identify the requirement it
validates and the expected result.

### Non-Functional Requirements
Every NFR needs a specific target. "Must be fast" is not a requirement.
"95th percentile response under 200ms" is. Only include NFRs relevant to
this specific feature, not product-wide NFRs from the PRD.

### User Stories
Reference by ID and title with a relative link. Do not duplicate story
content — the story file is the source of truth. If stories haven't been
written yet, list placeholders with `[TODO: create story]` and note it in
Open Questions.

### Edge Cases and Error Handling
Feature-level edge cases that span multiple stories. If an edge case is
specific to one story, it belongs in that story's file.

### Success Metrics
Feature-specific metrics, not product-level metrics from the PRD. How do
you know this specific feature is working as intended?

### Dependencies
Name specific feature IDs, external APIs, and PRD requirement numbers.
"Depends on auth" is too vague. "Depends on FEAT-002 (auth middleware)
and the OAuth2 provider API" is specific.

### Out of Scope
Each item should prevent a plausible scope question during implementation.
"Not a replacement for the database" is only useful if someone might think
it is.

## Quality Checklist

After drafting, verify every item. If any blocking check fails, revise before
committing.

### Blocking

- [ ] Overview links to a specific PRD requirement
- [ ] Ideal Future State is present for broad product-surface, workflow, IA, or documentation features
- [ ] Functional Areas is present when the feature spans multiple surfaces, workflows, user modes, or domain objects
- [ ] Similar domain objects are separated before requirements are written
- [ ] Functional requirements are grouped by area when a flat list would mix unrelated scopes
- [ ] Every functional requirement is testable
- [ ] Acceptance criteria cover the highest-risk requirements with observable examples
- [ ] Non-functional requirements have specific numeric targets
- [ ] User stories are referenced by ID (not duplicated inline)
- [ ] Dependencies name specific feature IDs and external systems
- [ ] No `[NEEDS CLARIFICATION]` markers remain

### Warning

- [ ] Problem statement quantifies the pain
- [ ] At least one feature-level edge case documented
- [ ] Success metrics are feature-specific (not product-level)
- [ ] Out of scope excludes something plausible
-Template
Show the template structure
---
ddx:
  id: FEAT-XXX
---

# Feature Specification: FEAT-XXX — [Feature Name]

**Feature ID**: FEAT-XXX
**Status**: [Draft | Specified | Approved]
**Priority**: [P0 | P1 | P2]
**Owner**: [Team/Person]
**Covered PRD Subsystem(s)**: [Subsystem name(s) from the PRD — normally exactly one]
**Covered PRD Requirements**: [FR-n, FR-m — the PRD FRs this feature owns]
**Cross-Subsystem Rationale**: [None — single subsystem. | If more than one subsystem
is listed above: the rationale that the cross-subsystem workflow IS the feature;
otherwise split it per the Decomposition test.]
<!-- reconcile-alignment reads these three fields: a feature spanning >1 subsystem
with no Cross-Subsystem Rationale is a mega-FEAT finding. -->

## Overview

[What this feature is and why it exists. 2-3 sentences connecting this feature
to a specific PRD requirement.]

## Ideal Future State

[Describe the future product behavior once this feature is working well. Focus
on what users can understand, decide, or accomplish. For broad product-surface,
workflow, IA, or documentation features, this section should come before the
problem framing so requirements are pulled toward the desired outcome instead
of only reacting to current pain.]

## Problem Statement

- **Current situation**: [What exists now — be specific]
- **Pain points**: [What is not working and for whom]
- **Desired outcome**: [What success looks like — measurable]

## Functional Areas

[For features with more than one surface or stage **within this one capability**,
map the subordinate areas before writing requirements. Areas are parts of one
capability — each fails the ship/cut/metric test on its own. Lists of roles,
lifecycle stages, or distinct domain objects are usually *separate features*, not
areas (apply the Decomposition test). Omit when the feature is a single narrow
capability.]

| Area | User question or job | Feature responsibility |
|------|----------------------|------------------------|
| [Area] | [What the user needs to know or do] | [What this feature must provide] |

## Requirements

### Functional Requirements by Area

[Each requirement should be testable. Group requirements by functional area
when the feature has multiple areas. Use stable prefixes that make the scope
clear, such as `HOME-01`, `TYPE-01`, `NAV-01`, or `FR-01` for narrow features.]

#### [Area Name]

[PREFIX-01]. [Requirement]
[PREFIX-02]. [Requirement]

### Acceptance Criteria

[Capture observable examples for the highest-risk or most important
requirements. Given/When/Then phrasing is allowed but not required. Do not
describe implementation steps.]

| Requirement | Scenario | Given | When | Then |
|-------------|----------|-------|------|------|
| [PREFIX-01] | [Observable case] | [Starting state] | [User/system action] | [Expected result] |

### Non-Functional Requirements

- **Performance**: [Specific target, e.g., "95th percentile response < 200ms"]
- **Security**: [Specific requirement, not "must be secure"]
- **Scalability**: [Specific target, e.g., "handles 10k concurrent users"]
- **Reliability**: [Specific target, e.g., "99.9% uptime"]

## User Stories

[List the user stories that implement this feature. Each story is a separate
file in `docs/helix/01-frame/user-stories/`. Reference by ID — do not
duplicate story content here.]

- [US-XXX — Story title](../user-stories/US-XXX-slug.md)
- [US-XXX — Story title](../user-stories/US-XXX-slug.md)

## Edge Cases and Error Handling

[Feature-level edge cases that span multiple stories. Story-specific edge
cases belong in the story file.]

- **[Condition]**: [Expected behavior]

## Success Metrics

[How do we know this feature is working? Metrics specific to this feature,
not the product-level metrics from the PRD.]

- [Metric with target]

## Constraints and Assumptions

- [Constraint or assumption specific to this feature]

## Dependencies

- **Other features**: [FEAT-XXX if this feature depends on another]
- **External services**: [APIs, libraries, or systems this feature requires]
- **PRD requirements**: [Which P0/P1/P2 requirements this addresses]

## Out of Scope

[What this feature explicitly does not cover. Each item should prevent a
plausible scope question.]

## Review Checklist

Use this checklist when reviewing a feature specification:

- [ ] Covered PRD Subsystem(s) and Requirements (`FR-n`) are listed; a feature spanning >1 subsystem carries an explicit cross-subsystem rationale (else split per the Decomposition test)
- [ ] Functional areas (if any) are subordinate parts of this one capability, not separate capabilities (each fails the ship/cut/metric test on its own)
- [ ] Overview connects this feature to a specific PRD requirement
- [ ] Ideal future state describes the desired user-visible outcome, not only current problems
- [ ] Problem statement describes what exists now and what is broken — not just what is wanted
- [ ] Functional areas are mapped when the feature spans multiple surfaces, workflows, or domain objects
- [ ] Requirements are grouped by functional area when a flat list would mix unrelated scopes
- [ ] Domain objects that sound similar are explicitly separated (for example, artifact instances vs artifact types)
- [ ] Every functional requirement is testable — you can write an assertion for it
- [ ] Acceptance criteria cover important happy paths, errors, and edge cases with observable outcomes
- [ ] Non-functional requirements have specific numeric targets, not "must be fast"
- [ ] Edge cases cover realistic failure scenarios, not just happy paths
- [ ] Success metrics are specific to this feature, not product-level metrics
- [ ] Dependencies reference real artifact IDs (FEAT-XXX, external APIs)
- [ ] Out of scope excludes things someone might reasonably assume are in scope
- [ ] No implementation details ("use X library", "create Y table") — specify WHAT not HOW
- [ ] Feature is consistent with governing PRD requirements
- [ ] No `[NEEDS CLARIFICATION]` markers remain unresolved for P0 features
+Generation prompt
Show the full generation prompt
# Feature Specification Generation Prompt

Create a feature specification that is precise enough to support design,
user story creation, and test planning. The feature spec owns FR-IDs,
functional areas, and the decomposition test — acceptance criteria live in
user-stories (see ADR-009) and must not be restated here.

## Storage Location

Store at: `docs/helix/01-frame/features/FEAT-NNN-<name>.md`

## Purpose

A feature spec is the **feature-level authority for behavior and boundaries**.
It translates PRD requirements into precise feature behavior, functional areas,
non-functional expectations, edge cases, and feature-specific success measures.
Acceptance criteria belong to user stories (ADR-009) and are not defined here.

It sits between the PRD (which defines product scope) and user stories (which
define vertical slices through the feature). The feature spec owns feature
behavior. User stories own user journeys. Solution and technical designs own
how the behavior will be built.

## Reference Anchors

Use these local resource summaries as grounding:

- `docs/resources/ibm-requirements-management.md` grounds traceable,
  prioritized, verifiable requirements.
- `docs/resources/cucumber-executable-specifications.md` grounds concrete
  examples as readable acceptance specifications without prescribing
  implementation or tooling.

## Active Concerns

For each concern selected in `docs/helix/01-frame/concerns.md`, apply its declared
`## Artifact Impact` (from `workflows/concerns/<name>/concern.md`) to THIS feature spec — realize the
FEAT-level obligations it names (usage-metering -> which actions are billable; multi-tenancy -> tenant-scoped ACs). A selected concern whose Artifact Impact names FEAT
but leaves no trace here is drift (reconcile-alignment Concern->Artifact Realization check).

## Key Principles

- **Future state before current pain** — describe the desired user-visible
  outcome before optimizing around today's broken surface. The problem statement
  explains why the change is needed; it should not be the only organizing frame.
- **Scope, not solution** — describe what the feature must do, not how to
  build it. Implementation details belong in design docs.
- **Behavior, not journey** — specify feature behavior and boundaries.
  Put end-to-end user flow narrative and acceptance criteria in user stories.
- **One feature, one capability** — a feature spec covers exactly one capability
  (≈ one PRD subsystem). If it covers two, split it; apply the Decomposition test
  below to decide. A functional *area* is a sub-part of one capability, not a
  second capability.
- **Functional areas before requirements** — when a feature spans multiple
  surfaces, stages, or domain objects *within the one capability*, name those
  areas before writing requirements and group requirements by area instead of
  producing one flat list. (Areas are subordinate parts of one capability — if an
  "area" would pass the Decomposition test as its own capability, it is a separate
  feature, not an area.)
- **Separate similar domain objects** — if readers might confuse two things,
  define them separately before requirements. For example, "Artifacts" are
  project-specific instances; "Artifact Types" are reusable methodology
  definitions.
- **Stories by reference** — list user story IDs, don't duplicate story
  content. Stories are separate files with their own lifecycle.
- **Testable requirements** — every functional requirement should be
  verifiable. If you can't describe how to test it, it's too vague.
- **Leave unknowns explicit** — use Open Questions at the bottom rather than
  inventing detail you don't have.

## Boundary Test

| If you are writing... | Put it in... |
|---|---|
| Product goals, personas, launch priority, or product-level metrics | PRD |
| Feature behavior, boundaries, and edge cases | Feature Specification |
| A vertical user journey through one or more feature requirements, with acceptance criteria | User Story |
| Component choices, data model, APIs, or implementation approach | Solution/Technical Design |
| Detailed test cases, fixtures, or automation strategy | Test Plan or Story Test Plan |
| Build sequencing and work slices | Implementation Plan |

## Decomposition — is it a FEAT or a functional area?

The brief decomposes into features at one granularity: **one feature per
capability**, anchored to the PRD's `### Subsystem:` groupings (~one subsystem →
one `FEAT-NNN`). Use these **layered tests** to place a candidate:

1. **Primary — ship / cut / metric.** A candidate is its own **feature** if all
   hold:
   - **Ship/cut:** it could be removed or deferred **without making another
     *named* capability incoherent** (it stands alone in the parking-lot).
   - **Metric:** it carries its own **feature-level product/user outcome** as a
     success metric — not a local counter (a button click or a row count is not a
     feature metric).
   If a candidate fails these — it cannot stand alone and has no outcome of its
   own — it is a **functional area** within a feature, not a feature.
2. **Tie-breaker — bounded context.** When ship/cut is genuinely ambiguous, split
   on bounded context / aggregate root: one feature per bounded context; areas are
   views/stages over the *same* aggregate.

**Anchor:** the PRD names the subsystems; each maps to ~one feature. A
multi-subsystem brief that produces a single mega-feature, or that produces zero
feature specs (PRD → stories directly), has skipped this tier — reconcile-alignment
flags both. A deliberately cross-subsystem feature (the workflow that spans them
*is* the feature) is allowed, but must say so explicitly in the template's
**Cross-Subsystem Rationale** field (the "Covered PRD Subsystem(s)" /
"Covered PRD Requirements" fields hold the subsystem names and FR IDs).
## Section-by-Section Guidance

### Overview
Connect this feature to a specific PRD requirement. "This feature implements
PRD P0-3" is better than "This feature improves the user experience."

### Ideal Future State
Describe the target state in user-visible terms. A good future state answers:

- What can the user understand, decide, or accomplish?
- What does the product surface make clear?
- How should the feature feel when it is working well?

For IA, documentation, onboarding, workflow, or product-surface features, this
section is mandatory. It should lead the spec toward the desired experience,
not merely away from the current failure mode.

### Problem Statement
Same standard as the PRD: describe the failure mode, not the absence of your
feature. Quantify where possible. Keep it subordinate to the future state; do
not let the spec become a list of current complaints.

### Functional Areas
Use this section whenever a feature has more than one surface, stage, or domain
object **within its one capability**. The area map should make clear what belongs
where before requirements are written. Areas are *subordinate parts* of the
feature — each fails the Decomposition test on its own (it cannot ship/cut
independently and has no feature-level outcome of its own).

Examples (areas *inside one capability*):

- CSV lead import → field mapping, validation, duplicate handling, confirmation
- Template editor → block palette, variable insertion, live preview, save/version
- Campaign scheduler → recipient selection, send-time rules, blackout handling

**Caution:** lists of *roles* ("Admin, Operator, Auditor"), *lifecycle stages*
("Intake, Planning, Execution, Review"), or *distinct domain objects* ("Leads,
Lists, Segments", "API, CLI, docs") are usually **separate features**, not areas
of one — each typically passes the Decomposition test as its own capability.
Apply the test before treating them as areas.

### Functional Requirements
Number each requirement for traceability. Group requirements by functional
area when the feature spans multiple areas. Use stable prefixes that make the
scope clear (`NAV-01`, `TYPE-01`, `ART-01`) or use plain `FR-01` for narrow
single-area features.

Each requirement should be independently testable. These are what the feature
must do — user stories describe how users interact with these capabilities.

If a requirement mentions two areas joined by "and", split it unless the
relationship between those areas is itself the requirement.

### Non-Functional Requirements
Every NFR needs a specific target. "Must be fast" is not a requirement.
"95th percentile response under 200ms" is. Only include NFRs relevant to
this specific feature, not product-wide NFRs from the PRD.

### User Stories
Reference by ID and title with a relative link. Do not duplicate story
content — the story file is the source of truth. If stories haven't been
written yet, list placeholders with `[TODO: create story]` and note it in
Open Questions.

### Edge Cases and Error Handling
Feature-level edge cases that span multiple stories. If an edge case is
specific to one story, it belongs in that story's file.

### Success Metrics
Feature-specific metrics, not product-level metrics from the PRD. How do
you know this specific feature is working as intended?

### Dependencies
Name specific feature IDs, external APIs, and PRD requirement numbers.
"Depends on auth" is too vague. "Depends on FEAT-002 (auth middleware)
and the OAuth2 provider API" is specific.

### Out of Scope
Each item should prevent a plausible scope question during implementation.
"Not a replacement for the database" is only useful if someone might think
it is.

## Quality Checklist

After drafting, verify every item. If any blocking check fails, revise before
committing.

### Blocking

- [ ] Overview links to a specific PRD requirement
- [ ] Ideal Future State is present for broad product-surface, workflow, IA, or documentation features
- [ ] Functional Areas is present when the feature spans multiple surfaces, workflows, user modes, or domain objects
- [ ] Similar domain objects are separated before requirements are written
- [ ] Functional requirements are grouped by area when a flat list would mix unrelated scopes
- [ ] Every functional requirement is testable
- [ ] Non-functional requirements have specific numeric targets
- [ ] User stories are referenced by ID (not duplicated inline)
- [ ] Dependencies name specific feature IDs and external systems
- [ ] No `[NEEDS CLARIFICATION]` markers remain

### Warning

- [ ] Problem statement quantifies the pain
- [ ] At least one feature-level edge case documented
- [ ] Success metrics are feature-specific (not product-level)
- [ ] Out of scope excludes something plausible
+Template
Show the template structure
---
ddx:
  id: FEAT-XXX
---

# Feature Specification: FEAT-XXX — [Feature Name]

**Feature ID**: FEAT-XXX
**Status**: [Draft | Specified | Approved]
**Priority**: [P0 | P1 | P2]
**Owner**: [Team/Person]
**Covered PRD Subsystem(s)**: [Subsystem name(s) from the PRD — normally exactly one]
**Covered PRD Requirements**: [FR-n, FR-m — the PRD FRs this feature owns]
**Cross-Subsystem Rationale**: [None — single subsystem. | If more than one subsystem
is listed above: the rationale that the cross-subsystem workflow IS the feature;
otherwise split it per the Decomposition test.]
<!-- reconcile-alignment reads these three fields: a feature spanning >1 subsystem
with no Cross-Subsystem Rationale is a mega-FEAT finding. -->

## Overview

[What this feature is and why it exists. 2-3 sentences connecting this feature
to a specific PRD requirement.]

## Ideal Future State

[Describe the future product behavior once this feature is working well. Focus
on what users can understand, decide, or accomplish. For broad product-surface,
workflow, IA, or documentation features, this section should come before the
problem framing so requirements are pulled toward the desired outcome instead
of only reacting to current pain.]

## Problem Statement

- **Current situation**: [What exists now — be specific]
- **Pain points**: [What is not working and for whom]
- **Desired outcome**: [What success looks like — measurable]

## Functional Areas

[For features with more than one surface or stage **within this one capability**,
map the subordinate areas before writing requirements. Areas are parts of one
capability — each fails the ship/cut/metric test on its own. Lists of roles,
lifecycle stages, or distinct domain objects are usually *separate features*, not
areas (apply the Decomposition test). Omit when the feature is a single narrow
capability.]

| Area | User question or job | Feature responsibility |
|------|----------------------|------------------------|
| [Area] | [What the user needs to know or do] | [What this feature must provide] |

## Requirements

### Functional Requirements by Area

[Each requirement should be testable. Group requirements by functional area
when the feature has multiple areas. Use stable prefixes that make the scope
clear, such as `HOME-01`, `TYPE-01`, `NAV-01`, or `FR-01` for narrow features.]

#### [Area Name]

[PREFIX-01]. [Requirement]
[PREFIX-02]. [Requirement]

### Non-Functional Requirements

- **Performance**: [Specific target, e.g., "95th percentile response < 200ms"]
- **Security**: [Specific requirement, not "must be secure"]
- **Scalability**: [Specific target, e.g., "handles 10k concurrent users"]
- **Reliability**: [Specific target, e.g., "99.9% uptime"]

## User Stories

[List the user stories that implement this feature. Each story is a separate
file in `docs/helix/01-frame/user-stories/`. Reference by ID — do not
duplicate story content here.]

- [US-XXX — Story title](../user-stories/US-XXX-slug.md)
- [US-XXX — Story title](../user-stories/US-XXX-slug.md)

## Edge Cases and Error Handling

[Feature-level edge cases that span multiple stories. Story-specific edge
cases belong in the story file.]

- **[Condition]**: [Expected behavior]

## Success Metrics

[How do we know this feature is working? Metrics specific to this feature,
not the product-level metrics from the PRD.]

- [Metric with target]

## Constraints and Assumptions

- [Constraint or assumption specific to this feature]

## Dependencies

- **Other features**: [FEAT-XXX if this feature depends on another]
- **External services**: [APIs, libraries, or systems this feature requires]
- **PRD requirements**: [Which P0/P1/P2 requirements this addresses]

## Out of Scope

[What this feature explicitly does not cover. Each item should prevent a
plausible scope question.]

## Review Checklist

Use this checklist when reviewing a feature specification:

- [ ] Covered PRD Subsystem(s) and Requirements (`FR-n`) are listed; a feature spanning >1 subsystem carries an explicit cross-subsystem rationale (else split per the Decomposition test)
- [ ] Functional areas (if any) are subordinate parts of this one capability, not separate capabilities (each fails the ship/cut/metric test on its own)
- [ ] Overview connects this feature to a specific PRD requirement
- [ ] Ideal future state describes the desired user-visible outcome, not only current problems
- [ ] Problem statement describes what exists now and what is broken — not just what is wanted
- [ ] Functional areas are mapped when the feature spans multiple surfaces, workflows, or domain objects
- [ ] Requirements are grouped by functional area when a flat list would mix unrelated scopes
- [ ] Domain objects that sound similar are explicitly separated (for example, artifact instances vs artifact types)
- [ ] Every functional requirement is testable — you can write an assertion for it
- [ ] Acceptance criteria are defined in the user stories that decompose this feature, not here (ADR-009)
- [ ] Non-functional requirements have specific numeric targets, not "must be fast"
- [ ] Edge cases cover realistic failure scenarios, not just happy paths
- [ ] Success metrics are specific to this feature, not product-level metrics
- [ ] Dependencies reference real artifact IDs (FEAT-XXX, external APIs)
- [ ] Out of scope excludes things someone might reasonably assume are in scope
- [ ] No implementation details ("use X library", "create Y table") — specify WHAT not HOW
- [ ] Feature is consistent with governing PRD requirements
- [ ] No `[NEEDS CLARIFICATION]` markers remain unresolved for P0 features
diff --git a/docs/website/content/artifact-types/frame/parking-lot.md b/docs/website/content/artifact-types/frame/parking-lot.md index b225f149..c9267989 100644 --- a/docs/website/content/artifact-types/frame/parking-lot.md +++ b/docs/website/content/artifact-types/frame/parking-lot.md @@ -116,7 +116,7 @@ CSV-first pilot scope. Default locationdocs/helix/parking-lot.md RequiresNone EnablesNone -Generation prompt
Show the full generation prompt
# Parking Lot Prompt
Capture deferred work that should not stay in the active path.

## Reference Anchors

Use this local resource summary as grounding:

- `docs/resources/atlassian-product-backlog.md` grounds visible deferred work,
  reprioritization, and closing items that will not be pursued.

## Focus
- Record the item, why it is deferred, and where it belongs next.
- Keep the entry short.
- Link to any relevant artifact or issue.
- Include a concrete revisit trigger; "later" is not a trigger.

## Role Boundary

The Parking Lot is not the backlog or tracker. It holds deferred or future work
that should remain findable without contaminating current scope. Active work
belongs in the Feature Registry and runtime work items. Rejected work should be closed
or cancelled, not parked forever.

## Completion Criteria
- Deferred items are easy to find later.
- Nothing active is buried here.
- Every item has source, rationale, owner, and revisit trigger.
-Template
Show the template structure
---
ddx:
  id: parking-lot
---

# Parking Lot (Deferred / Future Work)

## Policy
- Rejected items do not belong here; close or cancel them instead.
- Active work does not belong here; track it in the Feature Registry and DDx.
- Deferred items must include rationale and revisit trigger.
- Revisit triggers must be objective enough for another agent to evaluate.
- Any parked artifact must set `ddx.parking_lot: true` in its frontmatter.

## Deferred / Future Items

### [Item Title]
- **Type**: Deferred | Future
- **Artifact Type**: Feature Spec | User Story | ADR | Solution Design | Implementation Plan | Other
- **Source**: FEAT-XXX / US-XXX / ADR-XXX / external
- **Rationale**: [Why deferred / why future]
- **Impact if Omitted**: [Risk/impact]
- **Dependencies**: [Blocked by / prerequisites]
- **Revisit Trigger**: [What must happen before reconsidering]
- **Target Activity/Milestone**: [Activity or release]
- **Owner**: [Person/team responsible for review]
- **Last Reviewed**: [Date]

## Parked Artifacts (Links)

### [Artifact Title]
- **Artifact File**: [path to parked artifact]
- **Status**: Parking Lot (Deferred | Future)
+Generation prompt
Show the full generation prompt
# Parking Lot Prompt
Capture deferred work that should not stay in the active path.

## Reference Anchors

Use this local resource summary as grounding:

- `docs/resources/atlassian-product-backlog.md` grounds visible deferred work,
  reprioritization, and closing items that will not be pursued.

## Focus
- Record the item, why it is deferred, and where it belongs next.
- Keep the entry short.
- Link to any relevant artifact or issue.
- Include a concrete revisit trigger; "later" is not a trigger.

## Role Boundary

The Parking Lot is not the backlog or tracker. It holds deferred or future work
that should remain findable without contaminating current scope. Active work
belongs in the Feature Registry and runtime work items. Rejected work should be closed
or cancelled, not parked forever.

## Completion Criteria
- Deferred items are easy to find later.
- Nothing active is buried here.
- Every item has source, rationale, owner, and revisit trigger.

## Promotion to Feature Registry

When a parked entry's revisit trigger fires and the item is ready to enter the
active pipeline, follow the promotion procedure documented in the
[Feature Registry prompt](../feature-registry/prompt.md#promotion-from-parking-lot)
(per [ADR-010](../../../../../docs/helix/02-design/adr/ADR-010-feature-registry-parking-lot-handoff.md)).
+Template
Show the template structure
---
ddx:
  id: parking-lot
---

# Parking Lot (Deferred / Future Work)

## Purpose

[Why this parking lot exists and what it covers — scope of deferred / future
work tracked here without distorting current commitments.]

## Policy
- Rejected items do not belong here; close or cancel them instead.
- Active work does not belong here; track it in the Feature Registry and DDx.
- Deferred items must include rationale and revisit trigger.
- Revisit triggers must be objective enough for another agent to evaluate.
- Any parked artifact must set `ddx.parking_lot: true` in its frontmatter.

## Deferred / Future Items

### [Item Title]
- **Type**: Deferred | Future
- **Artifact Type**: Feature Spec | User Story | ADR | Solution Design | Implementation Plan | Other
- **Source**: FEAT-XXX / US-XXX / ADR-XXX / external
- **Rationale**: [Why deferred / why future]
- **Impact if Omitted**: [Risk/impact]
- **Dependencies**: [Blocked by / prerequisites]
- **Revisit Trigger**: [What must happen before reconsidering]
- **Target Activity/Milestone**: [Activity or release]
- **Owner**: [Person/team responsible for review]
- **Last Reviewed**: [Date]

## Parked Artifacts (Links)

### [Artifact Title]
- **Artifact File**: [path to parked artifact]
- **Status**: Parking Lot (Deferred | Future)
diff --git a/docs/website/content/artifact-types/frame/prd.md b/docs/website/content/artifact-types/frame/prd.md index 72690654..9d8dcd8f 100644 --- a/docs/website/content/artifact-types/frame/prd.md +++ b/docs/website/content/artifact-types/frame/prd.md @@ -16,6 +16,14 @@ requirements and boundaries. It sits between the product vision (which defines direction) and feature specs (which define feature-level detail). Every design decision and implementation choice should trace back to a PRD requirement. +**(kind: data)** When `kind: data`, the PRD is the **data-product-scope +authority for what data to build and why**. Its job is to translate business +intent into data-centric requirements: data sources, consumer personas, +quality contracts, technical constraints (catalog, schema, medallion layer), +and measurable success metrics. It sits between the Product Vision and the +data-architecture artifact. Every data pipeline design choice and quality +expectation should trace back to a Data PRD requirement. + ## Authoring guidance - **Problem first** — the problem section should make someone feel the pain @@ -275,7 +283,7 @@ unresolved deposits consistently leave a named owner and next action. InformsPrinciples
Feature Specification
User Stories
Feature Registry Referenced bySolution Design
Test Plan HELIX documentsdocs/helix/01-frame/prd.md -Generation prompt
Show the full generation prompt
# PRD Generation Prompt

Create a PRD that frames the problem, product scope, priorities, and success
criteria clearly enough that downstream feature specs, designs, tests, and
implementation work can trace back to it.

## Storage Location

Store at: `docs/helix/01-frame/prd.md`

## Purpose

The PRD is the **product-scope authority for what to build and why**. Its
unique job is to translate the Product Vision into prioritized, measurable
requirements and boundaries. It sits between the product vision (which defines
direction) and feature specs (which define feature-level detail). Every design
decision and implementation choice should trace back to a PRD requirement.

## Reference Anchors

Use these local resource summaries as grounding:

- `docs/resources/atlassian-prd.md` frames a PRD as shared understanding of purpose, behavior, user needs, assumptions, out-of-scope items, and success criteria.
- `docs/resources/aha-prd-template.md` supports concise cross-functional scope: what is being built, who it is for, and how it delivers value.
- `docs/resources/ibm-requirements-management.md` grounds measurable, prioritized, traceable requirements and validation/verification discipline.

## Key Principles

- **Problem first** — the problem section should make someone feel the pain
  before they see the solution.
- **Decision-oriented** — every section should help someone make a build/skip
  decision. If a section doesn't inform a decision, it's filler.
- **Testable requirements** — every P0 requirement should be verifiable. If
  you can't describe how to test it, it's too vague.
- **Traceable boundaries** — requirements should connect upward to the Product
  Vision and downward to feature specs, designs, tests, and build work.
- **Honest non-goals** — non-goals should exclude things someone might
  reasonably expect to be in scope. "Not a replacement for X" only matters if
  someone might assume it is.

## Stay in Your Lane

Product requirements are for product scope. If you find yourself writing about:

| This content | Belongs in |
|---|---|
| Market sizing, ROI, investment case | `00-discover/business-case.md` |
| Positioning, target market, long-horizon strategic success | `00-discover/product-vision.md` |
| Detailed feature behavior and edge cases | `01-frame/features/FEAT-*.md` |
| User journey phrasing independent of product-level requirements | `01-frame/user-stories.md` |
| Architecture choices or implementation approach | `02-design/` |
| Detailed test cases and fixtures | `03-test/` |
| Build sequencing and execution slices | `04-build/implementation-plan.md` |

## Section-by-Section Guidance

### Summary
Write this last. This section must work as a **standalone 1-pager**: what
we're building, who uses it, the problem, the solution approach, and the top
2-3 success metrics. Someone who reads only this section should understand the
product well enough to decide whether to invest time in the full PRD. Test:
could a new team member read this alone and explain the product to someone
else?

### Problem
Describe the failure mode, not the absence of your solution. "Users don't have
a X" is weak. "Users spend N hours/week doing Y manually because Z doesn't
exist, leading to W failures" is strong. Quantify where possible.

### Goals
Each goal should describe a state change, not an activity. "Build a dashboard"
is an activity. "Operators can see system health without SSH" is a state
change.

### Success Metrics
Every metric needs three things: what you're measuring, a numeric target, and
how you'll measure it. "User satisfaction" is not a metric. "NPS > 40 from
monthly survey of active users" is. Drop the Timeline column — success metrics
should define what success looks like, not when it happens.

### Non-Goals
Each non-goal should prevent scope creep on something plausible. "Not a
general-purpose AI" is only useful if someone might think it is. Test: would
someone on the team argue for including this? If not, it's not a useful
non-goal.

### Personas
Name them. Give them a role, goals, and pain points specific enough to
validate with a real person. "Alex the Developer" with generic goals is a
template, not a persona.

### Requirements (P0/P1/P2)
P0 = the product is broken without this. P1 = the product is weak without
this. P2 = the product is better with this. If you have more than 7 P0s,
you're not prioritizing.

Each requirement should be stable enough to trace into feature specs and tests.
If a requirement describes a screen, algorithm, API field, or implementation
sequence in detail, move that detail downstream and keep the PRD at product
scope.

### Functional Requirements
These are the detailed behavioral specs. Each one should be testable — someone
reading it should know how to write an acceptance test.

**Group requirements under named subsystems.** Organize FRs by subsystem (not by
priority) under canonical, parseable headings: `### Subsystem: <name>`. Each
`FR-n` belongs to **exactly one** subsystem. A subsystem is a cohesive capability
of the product — the unit that becomes a feature: **one subsystem maps to ~one
feature spec** (`FEAT-NNN`). This is the PRD↔FEAT boundary — the PRD owns
*breadth* (it names every subsystem and enumerates all `FR-n` + priorities); the
feature spec owns *depth* (one subsystem's behavior, ACs, edge cases). A
multi-subsystem product must not collapse into a single mega-feature, nor should
one subsystem fragment into many tiny features. (The feature spec's Decomposition
test resolves borderline cases; reconcile-alignment checks that every subsystem
maps to a feature.)

Give each functional requirement a **stable `FR-n` ID** (`FR-1`, `FR-2`, …). The
ID is the trace anchor: every `FR-n` must be covered by ≥1 user story, and
reconcile-alignment flags any `FR-n` with no story as a coverage gap. Number
sequentially and never renumber on edit — downstream stories reference the ID by
name.

### Acceptance Test Sketches
For each P0 requirement, write a concrete scenario: what the user does, what
input they provide, and what observable result they see. These aren't full test
cases — they're the minimum an implementer (human or agent) needs to verify
the requirement is met. An AI agent should be able to read a sketch and write
a passing test without asking clarifying questions.

### Technical Context
Name the stack, key dependencies with versions, API schemas, and platform
targets. Be specific enough that an implementer knows what to install and what
interfaces to code against. "React" is not enough — "React 18 with TypeScript
5.x and Vite 6" is. If there's an API schema (OpenAPI, GraphQL SDL), point to
it. This section exists because AI agents need concrete dependency and
interface information to produce correct implementations.

**Important**: This section records stack decisions — it does not make them.
Stack selection rationale belongs in ADRs (Architecture Decision Records). If
you're documenting a choice that doesn't have an ADR yet, note it in Open
Questions. If an existing ADR contradicts what you'd write here, the ADR
governs until it's superseded.

### Constraints
Name real constraints, not aspirational ones. "Must work on mobile" is a
constraint only if you'd otherwise skip it. Budget, compliance, and platform
constraints matter most.

### Assumptions
These are bets. When an assumption is wrong, the plan breaks. Name each one
so the team knows what to watch.

### Risks
Each risk needs a concrete mitigation, not "monitor closely." If the
mitigation is monitoring, say what you'll monitor and what triggers action.

### Open Questions
List unresolved items explicitly rather than leaving `[TBD]` markers
scattered through the document. Each question should name who can answer it
and what's blocked by it. This section is honest about what you don't know
yet — it's better to have a clear list of unknowns than a document that
pretends to be complete.

### Success Criteria
These are the acceptance criteria for the entire initiative. They should be
observable outcomes ("operators can do X without Y") not activities ("we
shipped Z").

## Quality Checklist

After drafting, verify every item. If any blocking check fails, revise before
committing.

### Blocking

- [ ] Problem section quantifies the pain or names a specific failure mode
- [ ] Every P0 requirement is testable (someone could write an acceptance test)
- [ ] Every P0 has an acceptance test sketch with inputs and expected outputs
- [ ] Success metrics have numeric targets and named measurement methods
- [ ] Requirements trace upward to the Product Vision and downward to downstream artifacts
- [ ] No `[TBD]`, `[TODO]`, or `[NEEDS CLARIFICATION]` markers in any section except Open Questions
- [ ] Non-goals exclude something a reasonable person might assume is in scope
- [ ] Personas are specific enough to validate with a real user

### Warning

- [ ] Summary works as a standalone 1-pager (problem, solution, metrics)
- [ ] Goals describe state changes, not activities
- [ ] Risk mitigations are concrete actions, not "monitor"
- [ ] P0 requirements number 7 or fewer
- [ ] Assumptions are falsifiable
- [ ] Functional requirements are grouped under canonical `### Subsystem: <name>` headings (each `FR-n` under exactly one subsystem); each subsystem is a capability that maps to ~one feature spec
- [ ] Each functional requirement carries a stable `FR-n` ID for downstream story traceability
- [ ] Technical Context names specific versions, not just library names
- [ ] Open Questions name who can answer and what's blocked
-Template
Show the template structure
---
ddx:
  id: prd
---

# Product Requirements Document

## Summary

[This section should work as a standalone 1-pager. Include: what we're
building, who uses it, what problem it solves, the solution approach, and the
top 2-3 success metrics. Write this last — it should be a distillation of the
full PRD, not an introduction. Someone who reads only this section should
understand the product well enough to decide whether to read the rest.]

## Problem and Goals

### Problem

[What is broken or missing? Who is affected? Be specific about the failure
mode — not "users struggle with X" but "users spend N hours per week doing X
because Y doesn't exist."]

### Goals

1. [Primary goal — what changes for users]
2. [Secondary goal]

### Success Metrics

| Metric | Target | Measurement Method |
|--------|--------|--------------------|
| [Metric] | [Numeric target] | [Named tool or process] |

### Non-Goals

[What we are explicitly not trying to achieve. Each non-goal should exclude
something a reasonable person might assume is in scope.]

Deferred items tracked in `docs/helix/parking-lot.md`.

## Users and Scope

### Primary Persona: [Name]

**Role**: [Job title/function]
**Goals**: [What they want to achieve]
**Pain Points**: [Current frustrations — specific enough to validate]

### Secondary Persona: [Name]

[Same structure]

## Requirements

Each requirement should trace to the Product Vision and be specific enough to
drive feature specs, designs, tests, and implementation work without embedding
the detailed design here.

### Must Have (P0)

1. [Core capability — what must be true for the product to be usable]

### Should Have (P1)

1. [Important feature — valuable but not blocking launch]

### Nice to Have (P2)

1. [Enhancement — improves experience but can be deferred]

## Functional Requirements

[Detailed behavioral requirements grouped under canonical `### Subsystem: <name>`
headings. Each requirement is testable, and each `FR-n` belongs to exactly one
subsystem. A subsystem is a cohesive product capability — the unit that maps to
~one feature spec (`FEAT-NNN`). The PRD owns breadth (all subsystems + `FR-n` +
priority); feature specs own each subsystem's depth.

Each functional requirement carries a **stable `FR-n` ID** (e.g. `FR-1`). The ID
survives edits so downstream artifacts trace to a specific requirement by name:
every `FR-n` must map to ≥1 user story `US-n`, and reconcile-alignment checks that
mapping (and that each subsystem maps to a feature) as a coverage floor. Number
them sequentially; do not renumber on edit.]

### Subsystem: [Name — a cohesive capability that becomes ~one FEAT]

- **FR-1** — [behavioral requirement, testable]
- **FR-2** — [behavioral requirement, testable]

### Subsystem: [Name]

- **FR-3** — [behavioral requirement, testable]

## Acceptance Test Sketches

[For each P0 requirement, describe a concrete scenario with inputs and
expected outputs. These aren't full test cases — they're the minimum needed
for an implementer (human or agent) to verify the requirement is met.]

| Requirement | Scenario | Input | Expected Output |
|-------------|----------|-------|-----------------|
| [P0 requirement] | [What the user does] | [Specific input or state] | [Observable result] |

## Technical Context

[Stack, key dependencies with versions, API schemas, and platform targets.
Be specific enough that an implementer knows what to install and what
interfaces to code against. This section records current stack decisions — it
does not make them. Stack selection rationale belongs in ADRs. If a choice
here isn't backed by an ADR yet, note it in Open Questions.]

- **Language/Runtime**: [e.g., TypeScript 5.x, Node 20+]
- **Key Libraries**: [e.g., React 18, Tailwind CSS 4]
- **Data/Storage**: [e.g., PostgreSQL 16, Redis 7]
- **APIs**: [e.g., OpenAPI spec at docs/api/v2.yaml]
- **Platform Targets**: [e.g., Linux, macOS; browser: Chrome/Firefox/Safari latest]

## Constraints, Assumptions, Dependencies

### Constraints

- **Technical**: [Platform or technology limits]
- **Business**: [Budget, timeline, resource limits]
- **Legal/Compliance**: [Regulatory requirements]

### Assumptions

- [Key assumptions — what must be true for this plan to work]

### Dependencies

- [External systems, teams, or artifacts this work depends on]

## Risks

| Risk | Probability | Impact | Mitigation |
|------|-------------|--------|------------|
| [Risk] | High/Med/Low | High/Med/Low | [Concrete strategy, not "monitor"] |

## Open Questions

[Unresolved items that need answers before or during implementation. Each
should name who can answer it and what's blocked by it. Prefer explicit
questions here over `[TBD]` markers scattered through the document.]

- [ ] [Question] — blocks [what], ask [who]

## Success Criteria

[What must be true to call the initiative successful. These should be
observable outcomes, not activities.]

## Review Checklist

Use this checklist when reviewing a PRD artifact:

- [ ] Summary works as a standalone 1-pager — someone can decide whether to read the rest
- [ ] Problem statement describes a specific failure mode with concrete cost
- [ ] Goals are outcomes, not activities ("users can X" not "we build Y")
- [ ] Success metrics have numeric targets and named measurement methods
- [ ] Non-goals exclude things a reasonable person might assume are in scope
- [ ] Personas have specific pain points, not generic descriptions
- [ ] P0 requirements are necessary for launch — removing any one makes the product unusable
- [ ] P1/P2 requirements are correctly prioritized relative to each other
- [ ] Every P0 requirement has an acceptance test sketch
- [ ] Requirements can trace upward to the Product Vision and downward to downstream artifacts
- [ ] Functional requirements are testable — each can be verified with specific inputs and expected outputs
- [ ] Each functional requirement carries a stable `FR-n` ID so user stories can trace to it by name
- [ ] Functional requirements are grouped under canonical `### Subsystem: <name>` headings, each `FR-n` under exactly one subsystem; each subsystem is a capability that maps to ~one feature spec
- [ ] Technical context names specific versions and interfaces, not vague technology areas
- [ ] Risks have concrete mitigations ("we do X"), not vague strategies ("we monitor")
- [ ] Open questions name who can answer and what is blocked
- [ ] No contradictions between requirements sections
- [ ] PRD is consistent with the governing product vision
+Generation prompt
Show the full generation prompt
# PRD Generation Prompt

Create a PRD that frames the problem, product scope, priorities, and success
criteria clearly enough that downstream feature specs, designs, tests, and
implementation work can trace back to it.

## Variant Selection (`kind`)

The PRD has two framing variants selected by the `kind` field in `meta.yml`:

- **`kind: product`** (default) — frames general product requirements.
- **`kind: data`** — frames a data product (pipeline, warehouse, data
  platform, or data service) with data sources, consumers, quality
  contracts, and platform context.

The shape of the artifact is unified; sections marked **(kind: data)** below
swap in the data-product framing when `kind: data`. Per ADR-008, both
framings share one role and one template — pick the variant that matches the
authored artifact and follow its conditional guidance.

## Storage Location

Store at: `docs/helix/01-frame/prd.md` (for either variant).

## Purpose

The PRD is the **product-scope authority for what to build and why**. Its
unique job is to translate the Product Vision into prioritized, measurable
requirements and boundaries. It sits between the product vision (which defines
direction) and feature specs (which define feature-level detail). Every design
decision and implementation choice should trace back to a PRD requirement.

**(kind: data)** When `kind: data`, the PRD is the **data-product-scope
authority for what data to build and why**. Its job is to translate business
intent into data-centric requirements: data sources, consumer personas,
quality contracts, technical constraints (catalog, schema, medallion layer),
and measurable success metrics. It sits between the Product Vision and the
data-architecture artifact. Every data pipeline design choice and quality
expectation should trace back to a Data PRD requirement.

## Reference Anchors

Use these local resource summaries as grounding:

- `docs/resources/atlassian-prd.md` frames a PRD as shared understanding of purpose, behavior, user needs, assumptions, out-of-scope items, and success criteria.
- `docs/resources/aha-prd-template.md` supports concise cross-functional scope: what is being built, who it is for, and how it delivers value.
- `docs/resources/ibm-requirements-management.md` grounds measurable, prioritized, traceable requirements and validation/verification discipline.

**(kind: data)** When `kind: data`, also use:

- `docs/resources/databricks-unity-catalog.md` grounds data governance through unified catalog hierarchies (metastore → catalog → schema → volume/table).
- `docs/resources/databricks-lakehouse-medallion-architecture.md` grounds medallion topology (Bronze/Silver/Gold) and layer responsibilities in a Lakehouse.
- `docs/resources/databricks-sdp.md` grounds Databricks Semantic Data Platform governance, lineage, and quality contracts through `EXPECT ... ON VIOLATION ...` clauses and SDP-aware pipeline patterns.

If you adopt this on another data platform, substitute Databricks concepts
with the platform equivalent (Snowflake DB/Schema/Table; BigQuery
Project/Dataset/Table; Snowpipe/Streaming Inserts for Auto Loader;
dbt/Great Expectations for SDP `EXPECT` clauses). The medallion pattern
applies universally.

## Key Principles

- **Problem first** — the problem section should make someone feel the pain
  before they see the solution.
- **Decision-oriented** — every section should help someone make a build/skip
  decision. If a section doesn't inform a decision, it's filler.
- **Testable requirements** — every P0 requirement should be verifiable. If
  you can't describe how to test it, it's too vague.
- **Traceable boundaries** — requirements should connect upward to the Product
  Vision and downward to feature specs, designs, tests, and build work.
- **Honest non-goals** — non-goals should exclude things someone might
  reasonably expect to be in scope. "Not a replacement for X" only matters if
  someone might assume it is.

## Stay in Your Lane

Product requirements are for product scope. If you find yourself writing about:

| This content | Belongs in |
|---|---|
| Market sizing, ROI, investment case | `00-discover/business-case.md` |
| Positioning, target market, long-horizon strategic success | `00-discover/product-vision.md` |
| Detailed feature behavior and edge cases | `01-frame/features/FEAT-*.md` |
| User journey phrasing independent of product-level requirements | `01-frame/user-stories.md` |
| Architecture choices or implementation approach | `02-design/` |
| Detailed test cases and fixtures | `03-test/` |
| Build sequencing and execution slices | `04-build/implementation-plan.md` |

## Section-by-Section Guidance

### Summary
Write this last. This section must work as a **standalone 1-pager**: what
we're building, who uses it, the problem, the solution approach, and the top
2-3 success metrics. Someone who reads only this section should understand the
product well enough to decide whether to invest time in the full PRD. Test:
could a new team member read this alone and explain the product to someone
else?

### Problem
Describe the failure mode, not the absence of your solution. "Users don't have
a X" is weak. "Users spend N hours/week doing Y manually because Z doesn't
exist, leading to W failures" is strong. Quantify where possible.

### Goals
Each goal should describe a state change, not an activity. "Build a dashboard"
is an activity. "Operators can see system health without SSH" is a state
change.

### Success Metrics
Every metric needs three things: what you're measuring, a numeric target, and
how you'll measure it. "User satisfaction" is not a metric. "NPS > 40 from
monthly survey of active users" is. Drop the Timeline column — success metrics
should define what success looks like, not when it happens.

**(kind: data)** When `kind: data`, frame metrics for the data product
itself: throughput (rows/day), latency (max age end-to-end), quality score
(percentage of expectations passing), cost per GB or per DBU, freshness-SLA
compliance, and consumer satisfaction. Each metric still needs a numeric
target and a named measurement method — e.g., "SLA compliance > 95% measured
by on-time delivery vs. promised refresh cadence."

### Non-Goals
Each non-goal should prevent scope creep on something plausible. "Not a
general-purpose AI" is only useful if someone might think it is. Test: would
someone on the team argue for including this? If not, it's not a useful
non-goal.

### Personas
Name them. Give them a role, goals, and pain points specific enough to
validate with a real person. "Alex the Developer" with generic goals is a
template, not a persona.

**(kind: data)** When `kind: data`, frame this section as **Data Consumers**
instead of personas. Name actual teams, systems, or roles that consume the
data, their use case (what decisions they make), their freshness/latency SLA,
the key dimensions they query, and their access level (row, column, full).
Add an inventory of upstream **Data Sources** in a parallel section: source
system, schema/table, owner, update frequency, quality baseline, and notes
on API limits or retry policy. Generic personas are insufficient for a data
product — the consumer and source tables drive every downstream design and
quality decision.

### Requirements (P0/P1/P2)
P0 = the product is broken without this. P1 = the product is weak without
this. P2 = the product is better with this. If you have more than 7 P0s,
you're not prioritizing.

Each requirement should be stable enough to trace into feature specs and tests.
If a requirement describes a screen, algorithm, API field, or implementation
sequence in detail, move that detail downstream and keep the PRD at product
scope.

### Functional Requirements
These are the detailed behavioral specs. Each one should be testable — someone
reading it should know how to write an acceptance test.

**(kind: data)** When `kind: data`, the functional requirements describe
**data behavior**: ingestion cadence, deduplication rules, transformation
contracts, freshness windows, schema-evolution policy, and consumer-facing
table or feed definitions. Add a **Data Quality Requirements** subsection
with quality dimensions (completeness, timeliness, accuracy, uniqueness)
each carrying a P0 threshold, a P1 threshold, a measurement method, and an
enforcement strategy (alert, reject, quarantine). Reference the
`data-quality-expectations` artifact for executable `EXPECT` clauses per
medallion layer; the PRD owns the requirement, the expectations artifact
owns the contract.

**Group requirements under named subsystems.** Organize FRs by subsystem (not by
priority) under canonical, parseable headings: `### Subsystem: <name>`. Each
`FR-n` belongs to **exactly one** subsystem. A subsystem is a cohesive capability
of the product — the unit that becomes a feature: **one subsystem maps to ~one
feature spec** (`FEAT-NNN`). This is the PRD↔FEAT boundary — the PRD owns
*breadth* (it names every subsystem and enumerates all `FR-n` + priorities); the
feature spec owns *depth* (one subsystem's behavior, ACs, edge cases). A
multi-subsystem product must not collapse into a single mega-feature, nor should
one subsystem fragment into many tiny features. (The feature spec's Decomposition
test resolves borderline cases; reconcile-alignment checks that every subsystem
maps to a feature.)

Give each functional requirement a **stable `FR-n` ID** (`FR-1`, `FR-2`, …). The
ID is the trace anchor: every `FR-n` must be covered by ≥1 user story, and
reconcile-alignment flags any `FR-n` with no story as a coverage gap. Number
sequentially and never renumber on edit — downstream stories reference the ID by
name.

### Acceptance Test Sketches
For each P0 requirement, write a concrete scenario: what the user does, what
input they provide, and what observable result they see. These aren't full test
cases — they're the minimum an implementer (human or agent) needs to verify
the requirement is met. An AI agent should be able to read a sketch and write
a passing test without asking clarifying questions.

### Technical Context
Name the stack, key dependencies with versions, API schemas, and platform
targets. Be specific enough that an implementer knows what to install and what
interfaces to code against. "React" is not enough — "React 18 with TypeScript
5.x and Vite 6" is. If there's an API schema (OpenAPI, GraphQL SDL), point to
it. This section exists because AI agents need concrete dependency and
interface information to produce correct implementations.

**Important**: This section records stack decisions — it does not make them.
Stack selection rationale belongs in ADRs (Architecture Decision Records). If
you're documenting a choice that doesn't have an ADR yet, note it in Open
Questions. If an existing ADR contradicts what you'd write here, the ADR
governs until it's superseded.

**(kind: data)** When `kind: data`, frame the technical context as the
**data-platform context**: target catalog and schema (e.g., `prod.customer_360`),
medallion layer strategy (Bronze/Silver/Gold scopes and responsibilities),
ingestion pattern (Auto Loader, Streaming Tables, batch), processing model
(streaming vs batch vs incremental), compute tier (all-purpose, jobs,
serverless), storage format (Delta, Parquet), DBU budget assumption, and
governance posture (data classification, retention policy, audit trail,
lineage). Pin the access-control model: row-level security, column masking,
and the catalog policies that enforce it. Same ADR discipline applies — the
PRD records platform decisions, ADRs justify them.

### Constraints
Name real constraints, not aspirational ones. "Must work on mobile" is a
constraint only if you'd otherwise skip it. Budget, compliance, and platform
constraints matter most.

### Assumptions
These are bets. When an assumption is wrong, the plan breaks. Name each one
so the team knows what to watch.

### Risks
Each risk needs a concrete mitigation, not "monitor closely." If the
mitigation is monitoring, say what you'll monitor and what triggers action.

### Open Questions
List unresolved items explicitly rather than leaving `[TBD]` markers
scattered through the document. Each question should name who can answer it
and what's blocked by it. This section is honest about what you don't know
yet — it's better to have a clear list of unknowns than a document that
pretends to be complete.

### Success Criteria
These are the acceptance criteria for the entire initiative. They should be
observable outcomes ("operators can do X without Y") not activities ("we
shipped Z").

## Quality Checklist

After drafting, verify every item. If any blocking check fails, revise before
committing.

### Blocking

- [ ] Problem section quantifies the pain or names a specific failure mode
- [ ] Every P0 requirement is testable (someone could write an acceptance test)
- [ ] Every P0 has an acceptance test sketch with inputs and expected outputs
- [ ] Success metrics have numeric targets and named measurement methods
- [ ] Requirements trace upward to the Product Vision and downward to downstream artifacts
- [ ] No `[TBD]`, `[TODO]`, or `[NEEDS CLARIFICATION]` markers in any section except Open Questions
- [ ] Non-goals exclude something a reasonable person might assume is in scope
- [ ] Personas are specific enough to validate with a real user

### Warning

- [ ] Summary works as a standalone 1-pager (problem, solution, metrics)
- [ ] Goals describe state changes, not activities
- [ ] Risk mitigations are concrete actions, not "monitor"
- [ ] P0 requirements number 7 or fewer
- [ ] Assumptions are falsifiable
- [ ] Functional requirements are grouped under canonical `### Subsystem: <name>` headings (each `FR-n` under exactly one subsystem); each subsystem is a capability that maps to ~one feature spec
- [ ] Each functional requirement carries a stable `FR-n` ID for downstream story traceability
- [ ] Technical Context names specific versions, not just library names
- [ ] Open Questions name who can answer and what's blocked
+Template
Show the template structure
---
ddx:
  id: prd
kind: product  # `product` (default) frames general product requirements; `data` frames a data product (pipeline, warehouse, data platform, or service). See ADR-008.
---

# Product Requirements Document

> **Variant guidance.** This template carries both the default `product`
> framing and a `data` framing. Sections marked **(kind: data)** apply when
> `kind: data` and replace the corresponding `product` framing above them.
> When `kind: product`, ignore the **(kind: data)** blocks. The shape of the
> document is the same; the framing is parameterized.

## Summary

[This section should work as a standalone 1-pager. Include: what we're
building, who uses it, what problem it solves, the solution approach, and the
top 2-3 success metrics. Write this last — it should be a distillation of the
full PRD, not an introduction. Someone who reads only this section should
understand the product well enough to decide whether to read the rest.

**(kind: data)** Frame this as a standalone 1-pager for the **data product**:
what data we are building, who consumes it, the business problem it solves,
the data solution approach (sources, medallion layer strategy, consumption
shape), and the top 2-3 success metrics (freshness, quality, cost).]

## Problem and Goals

### Problem

[What is broken or missing? Who is affected? Be specific about the failure
mode — not "users struggle with X" but "users spend N hours per week doing X
because Y doesn't exist."

**(kind: data)** Be specific about the data failure mode — not "users
struggle with reporting" but "sales analysts spend 4 hours per week
reconciling pipeline outputs with source systems because current freshness
is 24 hours and source data changes hourly."]

### Goals

1. [Primary goal — what changes for users]
2. [Secondary goal]

### Success Metrics

| Metric | Target | Measurement Method |
|--------|--------|--------------------|
| [Metric] | [Numeric target] | [Named tool or process] |

**(kind: data)** When `kind: data`, frame metrics for the data product
itself (throughput, latency, quality score, cost per GB). Include a baseline
and cadence column:

| Metric | Target | Baseline | Measurement Method | Cadence |
|--------|--------|----------|--------------------|---------|
| [Throughput] | [e.g., 1M rows/day] | [Current: 100K rows/day] | [COUNT(*) from production table] | Daily |
| [Latency] | [e.g., ≤1 hour end-to-end] | [Current: 4 hours] | [MAX(ingestion_timestamp) - MAX(source_timestamp)] | Hourly |
| [Quality Score] | [e.g., ≥98%] | [Current: 85%] | [Automated quality checks pass rate] | Daily |
| [Cost per GB] | [e.g., $0.05/GB/month] | [Current: $0.12/GB/month] | [DBU spend / data volume] | Monthly |

### Non-Goals

[What we are explicitly not trying to achieve. Each non-goal should exclude
something a reasonable person might assume is in scope.]

Deferred items tracked in `docs/helix/parking-lot.md`.

## Users and Scope

### Primary Persona: [Name]

**Role**: [Job title/function]
**Goals**: [What they want to achieve]
**Pain Points**: [Current frustrations — specific enough to validate]

### Secondary Persona: [Name]

[Same structure]

### (kind: data) Data Consumers

[When `kind: data`, replace the persona blocks with concrete data consumers.]

#### Primary Consumer: [Name/Role]

**Team**: [Data Engineering, Analytics, Product, Finance, etc.]
**Use Case**: [What they do with the data; what decision it informs]
**Frequency**: [Real-time, daily, weekly, ad-hoc]
**Key Tables/Feeds**: [Which outputs matter most]

#### Data Consumer Requirements Table

| Consumer | Use Case | Freshness SLA | Latency Tolerance | Key Dimensions | Access Level |
|----------|----------|---------------|-------------------|----------------|--------------|
| [Team] | [What they do] | [e.g., hourly] | [max delay] | [customer_id, product_id, ...] | [Row-level, Column-level, or Full] |

### (kind: data) Data Sources

[Inventory of upstream systems supplying this data product.]

| Source System | Schema / Table | Owner | Update Frequency | Quality Baseline | Notes |
|---------------|----------------|-------|------------------|------------------|-------|
| [e.g., Salesforce] | [e.g., Accounts, Opportunities] | [Team] | [hourly, daily, on-demand] | [% completeness, freshness] | [Data model version, API limits, retry policy] |

## Requirements

Each requirement should trace to the Product Vision and be specific enough to
drive feature specs, designs, tests, and implementation work without embedding
the detailed design here.

### Must Have (P0)

1. [Core capability — what must be true for the product to be usable]

### Should Have (P1)

1. [Important feature — valuable but not blocking launch]

### Nice to Have (P2)

1. [Enhancement — improves experience but can be deferred]

## Functional Requirements

[Detailed behavioral requirements grouped under canonical `### Subsystem: <name>`
headings. Each requirement is testable, and each `FR-n` belongs to exactly one
subsystem. A subsystem is a cohesive product capability — the unit that maps to
~one feature spec (`FEAT-NNN`). The PRD owns breadth (all subsystems + `FR-n` +
priority); feature specs own each subsystem's depth.

Each functional requirement carries a **stable `FR-n` ID** (e.g. `FR-1`). The ID
survives edits so downstream artifacts trace to a specific requirement by name:
every `FR-n` must map to ≥1 user story `US-n`, and reconcile-alignment checks that
mapping (and that each subsystem maps to a feature) as a coverage floor. Number
them sequentially; do not renumber on edit.]

### Subsystem: [Name — a cohesive capability that becomes ~one FEAT]

- **FR-1** — [behavioral requirement, testable]
- **FR-2** — [behavioral requirement, testable]

### Subsystem: [Name]

- **FR-3** — [behavioral requirement, testable]

### (kind: data) Data Quality Requirements

[When `kind: data`, add this subsection. Quality dimensions with numeric
thresholds and enforcement strategy. Reference `data-quality-expectations`
for executable `EXPECT` clauses per medallion layer.]

| Dimension | P0 Threshold | P1 Threshold | Measurement Method | Enforcement |
|-----------|--------------|--------------|--------------------|-------------|
| Completeness | [e.g., ≥99%] | [e.g., ≥95%] | [Count NULLs / total rows] | [Alert if below P0] |
| Timeliness | [e.g., ≤1 hour lag] | [e.g., ≤4 hour lag] | [MAX(ingestion_time) - MAX(source_time)] | [Reject data if exceeds P0] |
| Accuracy | [e.g., ≥98% match to source] | [e.g., ≥95% match] | [Row-count reconciliation + sample audit] | [Manual review + auto-reject if P0 fails] |
| Uniqueness | [e.g., PK has no duplicates] | [as P0] | [COUNT(*) = COUNT(DISTINCT PK)] | [Fail ingestion] |

## Acceptance Test Sketches

[For each P0 requirement, describe a concrete scenario with inputs and
expected outputs. These aren't full test cases — they're the minimum needed
for an implementer (human or agent) to verify the requirement is met.]

| Requirement | Scenario | Input | Expected Output |
|-------------|----------|-------|-----------------|
| [P0 requirement] | [What the user does] | [Specific input or state] | [Observable result] |

## Technical Context

[Stack, key dependencies with versions, API schemas, and platform targets.
Be specific enough that an implementer knows what to install and what
interfaces to code against. This section records current stack decisions — it
does not make them. Stack selection rationale belongs in ADRs. If a choice
here isn't backed by an ADR yet, note it in Open Questions.]

- **Language/Runtime**: [e.g., TypeScript 5.x, Node 20+]
- **Key Libraries**: [e.g., React 18, Tailwind CSS 4]
- **Data/Storage**: [e.g., PostgreSQL 16, Redis 7]
- **APIs**: [e.g., OpenAPI spec at docs/api/v2.yaml]
- **Platform Targets**: [e.g., Linux, macOS; browser: Chrome/Firefox/Safari latest]

### (kind: data) Data Platform Context

[When `kind: data`, replace the stack list above with platform context.]

- **Target Catalog**: [e.g., `prod`, `analytics`, or domain-specific catalog]
- **Target Schema**: [e.g., `customer_360`, `payment_events`]
- **Medallion Layers**: Bronze (raw), Silver (validated), Gold (business)
- **Access Control Model**: [UC policies, row-level security, column masking]

| Feature | Decision | Rationale |
|---------|----------|-----------|
| Ingestion Pattern | [Auto Loader, Streaming Tables, batch] | [Why this choice?] |
| Processing Model | [Streaming, Batch, Incremental] | [Freshness SLA and cost tradeoff] |
| Compute Tier | [All-purpose, Jobs, Serverless] | [Workload characteristics, cost model] |
| Storage Format | [Delta, Parquet, CSV] | [Durability, query performance needs] |
| DBU Budget (Monthly) | [Estimated spend] | [Based on row volume, freshness, complexity] |

- **Data Classification**: [Public, Internal, Sensitive, PII]
- **Retention Policy**: [e.g., Bronze: 7 days, Silver: 90 days, Gold: 2 years]
- **Audit Trail**: [Who accessed what, when, why]
- **Lineage Tracking**: [Table-to-table dependencies for impact analysis]

## Constraints, Assumptions, Dependencies

### Constraints

- **Technical**: [Platform or technology limits]
- **Business**: [Budget, timeline, resource limits]
- **Legal/Compliance**: [Regulatory requirements]

### Assumptions

- [Key assumptions — what must be true for this plan to work]

### Dependencies

- [External systems, teams, or artifacts this work depends on]

## Risks

| Risk | Probability | Impact | Mitigation |
|------|-------------|--------|------------|
| [Risk] | High/Med/Low | High/Med/Low | [Concrete strategy, not "monitor"] |

## Open Questions

[Unresolved items that need answers before or during implementation. Each
should name who can answer it and what's blocked by it. Prefer explicit
questions here over `[TBD]` markers scattered through the document.]

- [ ] [Question] — blocks [what], ask [who]

## Success Criteria

[What must be true to call the initiative successful. These should be
observable outcomes, not activities.]

## Review Checklist

Use this checklist when reviewing a PRD artifact:

- [ ] Summary works as a standalone 1-pager — someone can decide whether to read the rest
- [ ] Problem statement describes a specific failure mode with concrete cost
- [ ] Goals are outcomes, not activities ("users can X" not "we build Y")
- [ ] Success metrics have numeric targets and named measurement methods
- [ ] Non-goals exclude things a reasonable person might assume are in scope
- [ ] Personas have specific pain points, not generic descriptions
- [ ] P0 requirements are necessary for launch — removing any one makes the product unusable
- [ ] P1/P2 requirements are correctly prioritized relative to each other
- [ ] Every P0 requirement has an acceptance test sketch
- [ ] Requirements can trace upward to the Product Vision and downward to downstream artifacts
- [ ] Functional requirements are testable — each can be verified with specific inputs and expected outputs
- [ ] Each functional requirement carries a stable `FR-n` ID so user stories can trace to it by name
- [ ] Functional requirements are grouped under canonical `### Subsystem: <name>` headings, each `FR-n` under exactly one subsystem; each subsystem is a capability that maps to ~one feature spec
- [ ] Technical context names specific versions and interfaces, not vague technology areas
- [ ] Risks have concrete mitigations ("we do X"), not vague strategies ("we monitor")
- [ ] Open questions name who can answer and what is blocked
- [ ] No contradictions between requirements sections
- [ ] PRD is consistent with the governing product vision
diff --git a/docs/website/content/artifact-types/frame/security-requirements.md b/docs/website/content/artifact-types/frame/security-requirements.md index 67eace4c..70d3049b 100644 --- a/docs/website/content/artifact-types/frame/security-requirements.md +++ b/docs/website/content/artifact-types/frame/security-requirements.md @@ -178,7 +178,7 @@ NIST Privacy Framework as privacy-risk guidance. EnablesNone InformsSolution Design
Threat Model
Compliance Requirements Referenced byTest Plan -Generation prompt
Show the full generation prompt
# Security Requirements Generation Prompt
Document the security requirements the project must satisfy before design and build.

## Reference Anchors

Use these local resource summaries as grounding:

- `docs/resources/owasp-asvs.md` grounds application security requirements and
  verification expectations.
- `docs/resources/ftc-safeguards-rule.md` grounds financial customer
  information safeguards and applicability caveats.
- `docs/resources/nist-privacy-framework.md` grounds privacy risk management
  and data-processing controls.

## Focus
- Cover authentication, authorization, data protection, privacy, validation, and logging.
- Turn requirements into concrete controls and tests where possible.
- Keep compliance and risk notes brief but explicit.
- Make every requirement verifiable by design review, automated test, manual test, or audit evidence.

## Role Boundary

Security Requirements is not the Threat Model or Security Architecture. It
states what security outcomes and controls must be true. Threat Model explains
how the system can be attacked; Security Architecture explains where and how
controls are implemented.

## Completion Criteria
- Required controls are identified.
- Risks and assumptions are visible.
- The result is specific enough to guide design.
- Acceptance criteria are testable and traceable to controls, risks, or compliance obligations.
+Generation prompt
Show the full generation prompt
# Security Requirements Generation Prompt
Specify the testable acceptance criteria for the security controls the project
must satisfy before design and build.

## Traceability chain

This artifact's place in the security triangle: **controls -> testable
acceptance criteria**.
- See `compliance-requirements` for the regulations that motivate each control.
- See `threat-model` for the STRIDE categories and mitigation owners those
  controls answer to.

## Reference Anchors

Use these local resource summaries as grounding:

- `docs/resources/owasp-asvs.md` grounds application security requirements and
  verification expectations.
- `docs/resources/ftc-safeguards-rule.md` grounds financial customer
  information safeguards and applicability caveats.
- `docs/resources/nist-privacy-framework.md` grounds privacy risk management
  and data-processing controls.

## Focus
- Cover authentication, authorization, data protection, privacy, validation, and logging.
- Express each requirement as a testable acceptance criterion (design review,
  automated test, manual test, or audit evidence).
- For each control, cross-reference the originating obligation in
  `compliance-requirements` and the STRIDE owner in `threat-model`.
- Do not restate regulatory text or applicability analysis. Cross-reference
  `compliance-requirements`.
- Do not enumerate attacker behavior, data flows, or STRIDE categories.
  Cross-reference `threat-model`.

## Completion Criteria
- Required controls are identified and each has at least one testable
  acceptance criterion.
- Each criterion traces to a compliance obligation, a threat-model STRIDE
  owner, or both.
- The result is specific enough to guide design.
Template
Show the template structure
---
ddx:
  id: security-requirements
---

# Security Requirements

**Project**: [Project Name]
**Date**: [Creation Date]
**Security Champion**: [Name]

## Overview
[Brief description of security scope and key protection goals]

## Required Controls

### Authentication
- [Control and acceptance criteria]

### Authorization
- [Control and acceptance criteria]

### Data Protection
- [Control and acceptance criteria]

### Privacy
- [Control and acceptance criteria]

### Input Validation
- [Control and acceptance criteria]

### Logging and Audit
- [Control and acceptance criteria]

## Requirements Matrix

| ID | Requirement | Source | Risk Level | Verification |
|----|-------------|--------|------------|--------------|
| SEC-001 | [Requirement] | [Risk/compliance/ASVS] | High/Medium/Low | [Test/evidence] |

## Compliance Requirements
**Applicable Regulations**: [List]
**Applicable Standards**: [List]
- [Required control or obligation]

## Security Risks
### High-Risk Areas
1. **[Area]**: [Description and mitigation]

## Security Architecture Requirements
- [ ] Network segmentation
- [ ] Application security testing
- [ ] Dependency vulnerability scanning
- [ ] Server hardening
- [ ] Patch management
- [ ] Backup and recovery tested

## Security Testing Requirements
- [ ] Penetration testing
- [ ] Vulnerability assessments
- [ ] Security code review
- [ ] Automated security scanning

## Assumptions and Dependencies
- [Assumption or dependency]
diff --git a/docs/website/content/artifact-types/frame/stakeholder-map.md b/docs/website/content/artifact-types/frame/stakeholder-map.md index b91f14b7..fef08423 100644 --- a/docs/website/content/artifact-types/frame/stakeholder-map.md +++ b/docs/website/content/artifact-types/frame/stakeholder-map.md @@ -176,7 +176,7 @@ Low Power | Monitor | Keep Informed Default locationdocs/helix/01-frame/stakeholder-map.md RequiresNone EnablesNone -InformsPRD
Risk Register
Communication Plan +InformsPRD
Risk Register Generation prompt
Show the full generation prompt
# Stakeholder Map Generation Prompt
Map the stakeholders who matter most to the project and how to engage them.

## Reference Anchors

Use this local resource summary as grounding:

- `docs/resources/atlassian-raci-chart.md` grounds responsibility assignment,
  accountability, consultation, and informed stakeholders.

## Focus
- Identify primary and secondary stakeholders.
- Keep influence, interest, and communication needs clear.
- Note escalation paths without overexplaining them.
- Use roles when names or contact details are unknown. Do not invent people or emails.

## Role Boundary

Stakeholder Map is not a communication log or org chart. It names who has
influence, who needs input or updates, who is accountable for decisions, and
where unresolved decisions escalate.

## Completion Criteria
- Stakeholders are named and categorized.
- Communication is actionable.
- The map stays compact.
- Every critical decision has exactly one accountable role in the RACI matrix.
Template
Show the template structure
---
ddx:
  id: stakeholder-map
---

# Stakeholder Map

**Status**: [Draft | Review | Approved]
**Last Updated**: [Date]

## Primary Stakeholders (High Influence, High Interest)

### [Stakeholder Name/Role]
- **Role**: [Position/title]
- **Interest**: [What they care about]
- **Influence**: [Power to affect project]
- **Concerns**: [Main worries]
- **Success Criteria**: [What success looks like to them]
- **Communication**: [Preferred engagement method]
- **Contact Channel**: [Slack group / meeting / ticket queue / role alias]

## Secondary Stakeholders (Variable Influence/Interest)

### [Stakeholder Name/Role]
- **Role**:
- **Interest**:
- **Influence**:
- **Engagement Level**: [Inform | Consult | Collaborate]
- **Contact Channel**:

## RACI Matrix

| Activity/Decision | [Stakeholder 1] | [Stakeholder 2] | [Stakeholder 3] |
|-------------------|-----------------|-----------------|-----------------|
| Project Vision | A | R | C |
| Requirements | A | R | C |
| Technical Decisions | I | C | R/A |
| Budget Approval | A | C | I |
| Release Decision | A | R | C |

**R** = Responsible | **A** = Accountable | **C** = Consulted | **I** = Informed

## Power/Interest Grid

```
High Power  | Keep Satisfied      | Manage Closely
            | - [Stakeholder]     | - [Stakeholder]
            |---------------------|-------------------
Low Power   | Monitor             | Keep Informed
            | - [Stakeholder]     | - [Stakeholder]
            Low Interest          High Interest
```

## Communication Plan

| Stakeholder Group | Channel | Frequency | Content | Owner |
|-------------------|---------|-----------|---------|-------|
| [Group] | [Channel] | [Frequency] | [What] | [Who] |

### Escalation Path
1. [Team Lead] -> [PM] -> [Sponsor] -> [Executive]
diff --git a/docs/website/content/artifact-types/frame/threat-model.md b/docs/website/content/artifact-types/frame/threat-model.md index d25e8fd8..affd6644 100644 --- a/docs/website/content/artifact-types/frame/threat-model.md +++ b/docs/website/content/artifact-types/frame/threat-model.md @@ -217,7 +217,7 @@ ID prefix: S=Spoofing, T=Tampering, R=Repudiation, I=Information Disclosure, D=D EnablesNone InformsSolution Design
Test Plan Referenced byADR -Generation prompt
Show the full generation prompt
# Threat Modeling Prompt
Document the project threat model with enough detail to drive mitigations.

## Reference Anchors

Use these local resource summaries as grounding:

- `docs/resources/owasp-threat-modeling-cheat-sheet.md` grounds assets, data
  flows, trust boundaries, STRIDE, assumptions, and mitigation mapping.
- `docs/resources/owasp-asvs.md` grounds mapping threat mitigations into
  verifiable security controls.

## Focus
- Define boundaries, assets, and trust changes first.
- Analyze threats with STRIDE and map them to controls.
- Keep risk scoring and mitigation ownership explicit.
- Treat missing boundaries, unclear assets, or unstated assumptions as findings.

## Role Boundary

Threat Model is not Security Requirements or Security Architecture. It explains
what can go wrong and where. Security Requirements state what controls must be
true; Security Architecture places those controls in the design.

## Completion Criteria
- The threat surface is clear.
- Important threats are prioritized.
- Mitigations are concrete.
- High-risk threats have owners and verification hooks.
+Generation prompt
Show the full generation prompt
# Threat Modeling Prompt
Enumerate threats by STRIDE category and assign mitigation owners.

## Traceability chain

This artifact's place in the security triangle: **threats -> STRIDE +
mitigation owners**.
- See `security-requirements` for the testable controls that mitigate each
  threat.
- See `compliance-requirements` for the regulations those mitigations satisfy.

## Reference Anchors

Use these local resource summaries as grounding:

- `docs/resources/owasp-threat-modeling-cheat-sheet.md` grounds assets, data
  flows, trust boundaries, STRIDE, assumptions, and mitigation mapping.
- `docs/resources/owasp-asvs.md` grounds mapping threat mitigations into
  verifiable security controls.

## Focus
- Define boundaries, assets, and trust changes first.
- Analyze threats with STRIDE and assign each one a mitigation owner.
- Cross-reference the control(s) in `security-requirements` that mitigate the
  threat rather than restating control text.
- Keep risk scoring and mitigation ownership explicit.
- Treat missing boundaries, unclear assets, or unstated assumptions as findings.
- Do not author control acceptance criteria. Cross-reference
  `security-requirements`.
- Do not analyze regulatory applicability. Cross-reference
  `compliance-requirements`.

## Completion Criteria
- The threat surface is clear and threats are categorized by STRIDE.
- Each high-risk threat has a named mitigation owner and a cross-reference to
  the mitigating control in `security-requirements`.
- Important threats are prioritized.
Template
Show the template structure
---
ddx:
  id: threat-model
---

# Threat Model

**Project**: [Project Name]
**Date**: [Creation Date]

## Executive Summary

**System Overview**: [Brief description]
**Key Assets**: [Primary assets needing protection]
**Primary Threats**: [Top 3-5 threats]
**Risk Level**: [Critical/High/Medium/Low]

## System Description

### Boundaries and Components
**In Scope**: [Systems, components, data flows included]
**Out of Scope**: [What is not covered]
**Trust Boundaries**: [Where trust levels change]

### Components
| Component | Description | Trust Level |
|-----------|-------------|-------------|
| [Component] | [Description] | [Level] |

### Data Flows
- **External Sources**: [Data entering the system]
- **Internal Processing**: [How data moves within]
- **External Destinations**: [Where data exits]

## Assets

### Data Assets
| Asset | Classification | Confidentiality | Integrity | Availability |
|-------|---------------|-----------------|-----------|--------------|
| [Asset] | [Level] | [Criticality] | [Criticality] | [Criticality] |

### System Assets
| Asset | Criticality | Dependencies |
|-------|-------------|--------------|
| [Asset] | [Level] | [Dependencies] |

## STRIDE Threat Analysis

For each STRIDE category (Spoofing, Tampering, Repudiation, Information Disclosure, Denial of Service, Elevation of Privilege):

| ID | Threat | Impact | Likelihood | Risk | Mitigation |
|----|--------|--------|------------|------|------------|
| TM-X-001 | [Threat] | [Level] | [Level] | [Level] | [Control] |

ID prefix: S=Spoofing, T=Tampering, R=Repudiation, I=Information Disclosure, D=Denial of Service, E=Elevation of Privilege.

## Risk Assessment

**Scoring**: Impact (1-5) x Likelihood (1-5)
- **Critical (20-25)**: Immediate action required
- **High (15-19)**: Action within 30 days
- **Medium (10-14)**: Action within 90 days
- **Low (1-9)**: Monitor or accept

### Top Risks
| Risk ID | Threat | Impact | Likelihood | Score | Priority |
|---------|--------|--------|------------|-------|----------|
| [ID] | [Threat] | [1-5] | [1-5] | [Score] | [Level] |

## Mitigation Strategies

### [Risk ID] - [Title]
- **Controls**: [Preventive, detective, corrective actions]
- **Timeline**: [When to implement]
- **Owner**: [Who is responsible]
- **Verification**: [Security test, design review, or audit evidence]

## Security Controls Summary

- **Preventive**: [Authentication, authorization, encryption, input validation]
- **Detective**: [Logging, monitoring, intrusion detection]
- **Corrective**: [Incident response, backup/recovery, patching]

## Assumptions and Dependencies
- [List assumptions and external dependencies]
diff --git a/docs/website/content/artifact-types/frame/user-stories.md b/docs/website/content/artifact-types/frame/user-stories.md index eab172e4..86ea9ea2 100644 --- a/docs/website/content/artifact-types/frame/user-stories.md +++ b/docs/website/content/artifact-types/frame/user-stories.md @@ -53,7 +53,6 @@ committing. - [ ] "So that" names a measurable outcome, not a tautology - [ ] Walkthrough traces a complete path from trigger to outcome - [ ] Every acceptance criterion is independently testable (one Given/When/Then) -- [ ] Every acceptance criterion carries a stable `US--AC` ID - [ ] Test scenarios include concrete values, not placeholders - [ ] Story links to parent feature spec by ID - [ ] Story names the parent feature requirement IDs it exercises @@ -176,7 +175,7 @@ covered by follow-on stories. RequiresNone EnablesNone InformsSolution Design
Test Cases
Tracker Issues -Generation prompt
Show the full generation prompt
# User Story Generation Prompt

Create standalone user stories that serve as stable design artifacts — vertical
slices referenced throughout design, implementation, and testing.

## Storage Location

Store at: `docs/helix/01-frame/user-stories/US-NNN-<slug>.md` (one file per story)

## Purpose

User stories are **governing design artifacts**, not throwaway tickets. Each
story defines one persona's complete vertical journey through feature behavior
that is independently implementable and testable. Tracker issues reference
stories; stories don't reference tracker issues. Stories are more stable than
the implementation work items that fulfill them.

The feature spec owns behavior and boundaries. A user story owns a journey
through that behavior: who starts it, what they do, what the system shows, and
what outcome proves the slice works.

## Reference Anchors

Use these local resource summaries as grounding:

- `docs/resources/atlassian-user-stories.md` grounds persona-goal-value story
  framing and acceptance criteria.
- `docs/resources/cucumber-executable-specifications.md` grounds observable
  Given/When/Then acceptance criteria without requiring BDD tooling.

## Key Principles

- **One story, one vertical slice** — one persona completing one goal,
  demonstrable end-to-end in a single flow. If it can't be demonstrated
  end-to-end, it's not a story yet. The parent feature is the capability
  envelope; it has as many stories as it has distinct persona-goals. This is the
  FEAT↔story boundary.
- **One file per story** — each story is its own `US-NNN-<slug>.md` under
  `docs/helix/01-frame/user-stories/` (never a single monolithic
  `user-stories.md`); reconcile-alignment flags a monolithic stories file.
- **Stable reference** — stories will be referenced by multiple tracker issues
  across design, implementation, and testing. Write them to last.
- **Implementer-sufficient** — an implementer reading only this story and the
  parent feature spec should be able to build it without asking clarifying
  questions.
- **Test-first friendly** — acceptance criteria and test scenarios should be
  concrete enough to write tests before writing code.
- **Traceable to feature behavior** — each story should name the feature
  requirements it exercises. Do not invent behavior outside the parent feature
  spec.

## Boundary Test

| If you are writing... | Put it in... |
|---|---|
| Product-level scope, personas, priorities, or metrics | PRD |
| Complete feature behavior, functional areas, and edge cases | Feature Specification |
| One persona's journey through a feature slice | User Story |
| Component design, data model, API shape, or build approach | Solution/Technical Design |
| Detailed fixtures, test harnesses, or automation strategy | Story Test Plan |
| Work assignment, status, or execution notes | runtime work item or issue |

## Section-by-Section Guidance

### Story (As a / I want / So that)
The "As a" must name a specific persona from the PRD, not a generic role.
The "I want" must describe what the user does, not what the system does
internally. The "So that" must name a measurable outcome or business value —
"so that I can use the feature" is circular.

### Context
This is the background an implementer needs to make judgment calls. Why does
this story exist? What's the user's situation? Which parent feature
requirements does it exercise? What pain are they hitting?
2-4 sentences, not a paragraph of filler. Test: would removing this section
force the implementer to ask a question? If not, it's too generic.

### Walkthrough
A step-by-step journey through the vertical slice. Present tense, concrete
actions. This is not a flowchart — it's one specific path (the happy path)
from trigger to outcome. Branching and error cases go in Edge Cases.

Test: could a QA engineer use this walkthrough as a manual test script?

### Acceptance Criteria
Given/When/Then format. Each criterion must be independently testable — one
clear precondition, one action, one observable outcome. Avoid compound
criteria ("Given A and B and C, when D, then E and F and G"). Split those
into separate criteria.

Each criterion carries a **stable AC ID** of the form `US-<n>-AC<m>` (e.g.
`US-001-AC1`), where `<n>` is this story's number. The ID is stable across edits
so downstream artifacts reference a specific criterion by name. The story test
plan (STP) owns the matrix that maps each AC ID to the failing test(s) that
prove it — do **not** duplicate that matrix here; the story just defines the
criteria and their IDs. The project-level test plan (TP) aggregates strategy and
allocates criteria to test layers; it does not restate the per-AC matrix either
(FEAT-008 FR-6).

### Edge Cases
What happens when the user does something unexpected, inputs are invalid,
or the system is in an unusual state? Each edge case names the condition and
the expected behavior. Don't just list failure modes — specify what the system
should do.

### Test Scenarios
Concrete input/output pairs. An implementer should be able to copy these into
a test file with minimal modification. Include the happy path and at least one
edge case from the section above. Name specific values, not placeholders.

### Dependencies
Name other stories this one depends on (by ID), the parent feature spec,
and any external systems or APIs. If another story must be done first, say so.

### Traceability
Name the parent feature requirement IDs that the story exercises. If the story
needs behavior that is not in the feature spec, update the feature spec first.

Also name the **PRD functional requirement(s) `FR-n`** this story covers.
**Every PRD `FR-n` must map to ≥1 user story** — this is a coverage floor that
reconcile-alignment checks; a `FR-n` with no story is a blocking gap. A single
story may cover more than one `FR-n`, but **do not bundle unrelated `FR-n`s into
one story without recorded justification** — unrelated requirements belong in
separate vertical slices so each can be tested independently.

### Out of Scope
What this story explicitly does not cover. Each item should exclude something
an implementer might reasonably try to include. This prevents scope creep
during implementation.

## Quality Checklist

After drafting, verify every item. If any blocking check fails, revise before
committing.

### Blocking

- [ ] Story names a specific persona from the PRD (not a generic role)
- [ ] "I want" describes a user action, not a system behavior
- [ ] "So that" names a measurable outcome, not a tautology
- [ ] Walkthrough traces a complete path from trigger to outcome
- [ ] Every acceptance criterion is independently testable (one Given/When/Then)
- [ ] Every acceptance criterion carries a stable `US-<n>-AC<m>` ID
- [ ] Test scenarios include concrete values, not placeholders
- [ ] Story links to parent feature spec by ID
- [ ] Story names the parent feature requirement IDs it exercises
- [ ] Story names the PRD `FR-n` it covers; bundled unrelated `FR-n`s carry recorded justification

### Warning

- [ ] Context would be missed if removed (not generic filler)
- [ ] At least one edge case is documented
- [ ] Test scenarios cover both happy path and at least one edge case
- [ ] Out of scope excludes something plausible
- [ ] No compound acceptance criteria (split into separate items)
- [ ] Story does not invent behavior outside the parent feature spec
-Template
Show the template structure
---
ddx:
  id: US-XXX
---

# US-XXX: [Story Title]

**Feature**: [FEAT-XXX — Feature Name]
**Feature Requirements**: [REQ-01, REQ-02]
**PRD Requirements**: [FR-n — the PRD functional requirement(s) this story covers]
**Priority**: [P0 | P1 | P2]
**Status**: [Draft | Review | Approved]

## Story

**As a** [specific user type from PRD personas]
**I want** [specific functionality — what the user does, not what the system does]
**So that** [measurable business value or user outcome]

## Context

[Why this story matters. What's the user's situation before this works? What
problem are they hitting? Which parent feature requirements does this story
exercise? This should be 2-4 sentences that give an implementer enough
background to make judgment calls without asking.]

## Walkthrough

[Step-by-step description of the user's journey through this slice. Write in
present tense. Name concrete actions and system responses. This is the
vertical slice — it should cover one complete path from trigger to outcome.]

1. User [action]
2. System [response]
3. User [action]
4. System [response — the outcome]

## Acceptance Criteria

[Each criterion must be testable and carry a **stable AC ID** of the form
`US-<n>-AC<m>` (matching this story's number). Use Given/When/Then format — one
precondition, one action, one observable outcome. The ID is stable: it survives
edits so the story test plan and tests can reference a specific criterion by
name. An implementer should be able to write a passing test from each criterion
alone.]

- [ ] **US-XXX-AC1** — Given [specific precondition], when [specific action], then [observable outcome]
- [ ] **US-XXX-AC2** — Given [specific precondition], when [specific action], then [observable outcome]

## Edge Cases

[What happens when things go wrong or inputs are unexpected? Each edge case
should name the condition and the expected system behavior.]

- **[Condition]**: [Expected behavior]
- **[Condition]**: [Expected behavior]

## Test Scenarios

[Concrete input/output pairs for the acceptance criteria. An implementer
should be able to copy these into a test file.]

| Scenario | AC ID | Input / State | Action | Expected Result |
|----------|-------|---------------|--------|-----------------|
| Happy path | US-XXX-AC1 | [specific state] | [specific action] | [specific result] |
| [Edge case] | US-XXX-AC2 | [specific state] | [specific action] | [specific result] |

## Dependencies

- **Stories**: [US-XXX if this story depends on another being done first]
- **Feature Spec**: [FEAT-XXX]
- **Feature Requirements**: [REQ-01, REQ-02]
- **PRD Requirements**: [FR-n — PRD functional requirement(s) this story covers]
- **External**: [APIs, services, or data this story requires]

## Out of Scope

[What this story explicitly does not cover, to prevent scope creep during
implementation.]

## Review Checklist

Use this checklist when reviewing a user story:

- [ ] Stored as its own file `US-NNN-<slug>.md` (one file per story — never a single monolithic `user-stories.md`)
- [ ] Covers one persona completing one goal, demonstrable end-to-end in a single flow
- [ ] Links to its parent `FEAT-NNN` and names the PRD `FR-n` it covers
- [ ] Every acceptance criterion is independently testable and carries a stable `US-NNN-ACm` ID
- [ ] Walkthrough traces a complete path from trigger to outcome; at least one edge case documented
+Generation prompt
Show the full generation prompt
# User Story Generation Prompt

Create standalone user stories that serve as stable design artifacts — vertical
slices referenced throughout design, implementation, and testing.

## Storage Location

Store at: `docs/helix/01-frame/user-stories/US-NNN-<slug>.md` (one file per story)

## Purpose

User stories are **governing design artifacts**, not throwaway tickets. Each
story defines one persona's complete vertical journey through feature behavior
that is independently implementable and testable. Tracker issues reference
stories; stories don't reference tracker issues. Stories are more stable than
the implementation work items that fulfill them.

The feature spec owns behavior and boundaries. A user story owns a journey
through that behavior: who starts it, what they do, what the system shows, and
what outcome proves the slice works.

## Reference Anchors

Use these local resource summaries as grounding:

- `docs/resources/atlassian-user-stories.md` grounds persona-goal-value story
  framing and acceptance criteria.
- `docs/resources/cucumber-executable-specifications.md` grounds observable
  Given/When/Then acceptance criteria without requiring BDD tooling.

## Key Principles

- **One story, one vertical slice** — one persona completing one goal,
  demonstrable end-to-end in a single flow. If it can't be demonstrated
  end-to-end, it's not a story yet. The parent feature is the capability
  envelope; it has as many stories as it has distinct persona-goals. This is the
  FEAT↔story boundary.
- **One file per story** — each story is its own `US-NNN-<slug>.md` under
  `docs/helix/01-frame/user-stories/` (never a single monolithic
  `user-stories.md`); reconcile-alignment flags a monolithic stories file.
- **Stable reference** — stories will be referenced by multiple tracker issues
  across design, implementation, and testing. Write them to last.
- **Implementer-sufficient** — an implementer reading only this story and the
  parent feature spec should be able to build it without asking clarifying
  questions.
- **Test-first friendly** — acceptance criteria and test scenarios should be
  concrete enough to write tests before writing code.
- **Traceable to feature behavior** — each story should name the feature
  requirements it exercises. Do not invent behavior outside the parent feature
  spec.

## Boundary Test

| If you are writing... | Put it in... |
|---|---|
| Product-level scope, personas, priorities, or metrics | PRD |
| Complete feature behavior, functional areas, and edge cases | Feature Specification |
| One persona's journey through a feature slice | User Story |
| Component design, data model, API shape, or build approach | Solution/Technical Design |
| Detailed fixtures, test harnesses, or automation strategy | Story Test Plan |
| Work assignment, status, or execution notes | runtime work item or issue |

## Section-by-Section Guidance

### Story (As a / I want / So that)
The "As a" must name a specific persona from the PRD, not a generic role.
The "I want" must describe what the user does, not what the system does
internally. The "So that" must name a measurable outcome or business value —
"so that I can use the feature" is circular.

### Context
This is the background an implementer needs to make judgment calls. Why does
this story exist? What's the user's situation? Which parent feature
requirements does it exercise? What pain are they hitting?
2-4 sentences, not a paragraph of filler. Test: would removing this section
force the implementer to ask a question? If not, it's too generic.

### Walkthrough
A step-by-step journey through the vertical slice. Present tense, concrete
actions. This is not a flowchart — it's one specific path (the happy path)
from trigger to outcome. Branching and error cases go in Edge Cases.

Test: could a QA engineer use this walkthrough as a manual test script?

### Acceptance Criteria
Given/When/Then format. Each criterion must be independently testable — one
clear precondition, one action, one observable outcome. Avoid compound
criteria ("Given A and B and C, when D, then E and F and G"). Split those
into separate criteria.

Each criterion carries a **stable AC ID** of the form `US-<n>-AC<m>` (e.g.
`US-001-AC1`), where `<n>` is this story's number. The ID is stable across edits
so downstream artifacts reference a specific criterion by name. The story test
plan (STP) owns the matrix that maps each AC ID to the failing test(s) that
prove it — do **not** duplicate that matrix here; the story just defines the
criteria and their IDs. The project-level test plan (TP) aggregates strategy and
allocates criteria to test layers; it does not restate the per-AC matrix either
(FEAT-008 FR-6).

### Edge Cases
What happens when the user does something unexpected, inputs are invalid,
or the system is in an unusual state? Each edge case names the condition and
the expected behavior. Don't just list failure modes — specify what the system
should do.

### Test Scenarios
Concrete input/output pairs. An implementer should be able to copy these into
a test file with minimal modification. Include the happy path and at least one
edge case from the section above. Name specific values, not placeholders.

### Dependencies
Name other stories this one depends on (by ID), the parent feature spec,
and any external systems or APIs. If another story must be done first, say so.

### Traceability
Name the parent feature requirement IDs that the story exercises. If the story
needs behavior that is not in the feature spec, update the feature spec first.

Also name the **PRD functional requirement(s) `FR-n`** this story covers.
**Every PRD `FR-n` must map to ≥1 user story** — this is a coverage floor that
reconcile-alignment checks; a `FR-n` with no story is a blocking gap. A single
story may cover more than one `FR-n`, but **do not bundle unrelated `FR-n`s into
one story without recorded justification** — unrelated requirements belong in
separate vertical slices so each can be tested independently.

### Out of Scope
What this story explicitly does not cover. Each item should exclude something
an implementer might reasonably try to include. This prevents scope creep
during implementation.

## Quality Checklist

After drafting, verify every item. If any blocking check fails, revise before
committing.

### Blocking

- [ ] Story names a specific persona from the PRD (not a generic role)
- [ ] "I want" describes a user action, not a system behavior
- [ ] "So that" names a measurable outcome, not a tautology
- [ ] Walkthrough traces a complete path from trigger to outcome
- [ ] Every acceptance criterion is independently testable (one Given/When/Then)
- [ ] Test scenarios include concrete values, not placeholders
- [ ] Story links to parent feature spec by ID
- [ ] Story names the parent feature requirement IDs it exercises
- [ ] Story names the PRD `FR-n` it covers; bundled unrelated `FR-n`s carry recorded justification

### Warning

- [ ] Context would be missed if removed (not generic filler)
- [ ] At least one edge case is documented
- [ ] Test scenarios cover both happy path and at least one edge case
- [ ] Out of scope excludes something plausible
- [ ] No compound acceptance criteria (split into separate items)
- [ ] Story does not invent behavior outside the parent feature spec
+Template
Show the template structure
---
ddx:
  id: US-XXX
---

# US-XXX: [Story Title]

**Feature**: [FEAT-XXX — Feature Name]
**Feature Requirements**: [REQ-01, REQ-02]
**PRD Requirements**: [FR-n — the PRD functional requirement(s) this story covers]
**Priority**: [P0 | P1 | P2]
**Status**: [Draft | Review | Approved]

## Story

**As a** [specific user type from PRD personas]
**I want** [specific functionality — what the user does, not what the system does]
**So that** [measurable business value or user outcome]

## Context

[Why this story matters. What's the user's situation before this works? What
problem are they hitting? Which parent feature requirements does this story
exercise? This should be 2-4 sentences that give an implementer enough
background to make judgment calls without asking.]

## Walkthrough

[Step-by-step description of the user's journey through this slice. Write in
present tense. Name concrete actions and system responses. This is the
vertical slice — it should cover one complete path from trigger to outcome.]

1. User [action]
2. System [response]
3. User [action]
4. System [response — the outcome]

## Acceptance Criteria

[See the prompt's Acceptance Criteria guidance for the canonical AC-ID rule.]

- [ ] **US-XXX-AC1** — Given [specific precondition], when [specific action], then [observable outcome]
- [ ] **US-XXX-AC2** — Given [specific precondition], when [specific action], then [observable outcome]

## Edge Cases

[What happens when things go wrong or inputs are unexpected? Each edge case
should name the condition and the expected system behavior.]

- **[Condition]**: [Expected behavior]
- **[Condition]**: [Expected behavior]

## Test Scenarios

[Concrete input/output pairs for the acceptance criteria. An implementer
should be able to copy these into a test file.]

| Scenario | AC ID | Input / State | Action | Expected Result |
|----------|-------|---------------|--------|-----------------|
| Happy path | US-XXX-AC1 | [specific state] | [specific action] | [specific result] |
| [Edge case] | US-XXX-AC2 | [specific state] | [specific action] | [specific result] |

## Dependencies

- **Stories**: [US-XXX if this story depends on another being done first]
- **Feature Spec**: [FEAT-XXX]
- **Feature Requirements**: [REQ-01, REQ-02]
- **PRD Requirements**: [FR-n — PRD functional requirement(s) this story covers]
- **External**: [APIs, services, or data this story requires]

## Out of Scope

[What this story explicitly does not cover, to prevent scope creep during
implementation.]

## Review Checklist

Use this checklist when reviewing a user story:

- [ ] Stored as its own file `US-NNN-<slug>.md` (one file per story — never a single monolithic `user-stories.md`)
- [ ] Covers one persona completing one goal, demonstrable end-to-end in a single flow
- [ ] Links to its parent `FEAT-NNN` and names the PRD `FR-n` it covers
- [ ] Every acceptance criterion is independently testable and carries a stable `US-NNN-ACm` ID
- [ ] Walkthrough traces a complete path from trigger to outcome; at least one edge case documented
diff --git a/docs/website/content/artifact-types/iterate/improvement-backlog.md b/docs/website/content/artifact-types/iterate/improvement-backlog.md index 18ea9ce6..1a85161d 100644 --- a/docs/website/content/artifact-types/iterate/improvement-backlog.md +++ b/docs/website/content/artifact-types/iterate/improvement-backlog.md @@ -15,10 +15,14 @@ unique job is to convert metrics, feedback, incidents, and retrospective learnings into ordered improvement candidates with evidence, rationale, tracker or explicit follow-up targets, and a next-iteration selection. -It is not the live tracker. DDx or another runtime owns issue status, assignees, -and execution history. This artifact explains what should compete for attention +It is not the live tracker. The runtime owns issue status, assignees, and +execution history. This artifact explains what should compete for attention next and why. +For how this artifact relates to metric definitions, the metrics dashboard, +and security-metrics, see the "Metric Four-Way Slice" section of +`workflows/activities/06-iterate/README.md`. + ## Example
@@ -88,7 +92,7 @@ pilot-readiness review EnablesNone InformsTracker Issues HELIX documentsdocs/helix/06-iterate/improvement-backlog.md -Generation prompt
Show the full generation prompt
# Improvement Backlog Generation Prompt

Document the prioritized improvement inventory produced from iteration learnings.

## Purpose

Improvement Backlog is the **iteration follow-up prioritization artifact**. Its
unique job is to convert metrics, feedback, incidents, and retrospective
learnings into ordered improvement candidates with evidence, rationale, tracker
or explicit follow-up targets, and a next-iteration selection.

It is not the live tracker. DDx or another runtime owns issue status, assignees,
and execution history. This artifact explains what should compete for attention
next and why.

## Reference Anchors

Use this local resource summary as grounding:

- `docs/resources/intercom-rice-prioritization.md` grounds evidence-backed
  ranking by impact, confidence, effort, and reach.

## Focus
- Turn the current iteration's learnings into a ranked list of follow-up work.
- Prefer concrete tracker-backed items over vague TODOs.
- Use metrics, feedback, and retrospective findings as evidence.
- Make the next selection obvious by sorting by priority and impact.
- Link each item to the relevant work item, report, or supporting artifact.

## Boundary Test

| If you are writing... | Put it in... |
|---|---|
| Measurement interpretation for this iteration | Metrics Dashboard |
| Prioritized follow-up candidates and selection rationale | Improvement Backlog |
| Live issue status, assignee, and execution history | runtime work item or issue |
| New product requirements or design changes | Appropriate upstream artifact |

## Completion Criteria
- The inventory is prioritized.
- Every item has an evidence source.
- The next iteration candidates are explicit.
+Generation prompt
Show the full generation prompt
# Improvement Backlog Generation Prompt

Document the prioritized improvement inventory produced from iteration learnings.

## Purpose

Improvement Backlog is the **iteration follow-up prioritization artifact**. Its
unique job is to convert metrics, feedback, incidents, and retrospective
learnings into ordered improvement candidates with evidence, rationale, tracker
or explicit follow-up targets, and a next-iteration selection.

It is not the live tracker. The runtime owns issue status, assignees, and
execution history. This artifact explains what should compete for attention
next and why.

For how this artifact relates to metric definitions, the metrics dashboard,
and security-metrics, see the "Metric Four-Way Slice" section of
`workflows/activities/06-iterate/README.md`.

## Reference Anchors

Use this local resource summary as grounding:

- `docs/resources/intercom-rice-prioritization.md` grounds evidence-backed
  ranking by impact, confidence, effort, and reach.

## Focus
- Turn the current iteration's learnings into a ranked list of follow-up work.
- Prefer concrete tracker-backed items over vague TODOs.
- Use metrics, feedback, and retrospective findings as evidence.
- Make the next selection obvious by sorting by priority and impact.
- Link each item to the relevant work item, report, or supporting artifact.

## Boundary Test

| If you are writing... | Put it in... |
|---|---|
| Measurement interpretation for this iteration | Metrics Dashboard |
| Prioritized follow-up candidates and selection rationale | Improvement Backlog |
| Live issue status, assignee, and execution history | runtime work item or issue |
| New product requirements or design changes | Appropriate upstream artifact |

## Completion Criteria
- The inventory is prioritized.
- Every item has an evidence source.
- The next iteration candidates are explicit.
Template
Show the template structure
---
ddx:
  id: improvement-backlog
---

# Improvement Backlog

**Iteration**: [iteration or release]
**Source Learnings**: [metrics, feedback, retrospective, incident review]

## Prioritization Rules

- [Rule for ordering work]
- [Rule for handling safety or risk]
- [How confidence and effort affect ordering]

## Backlog Items

| Rank | Priority | Item | Evidence | Tracker or Follow-Up Target | Why Now | Confidence | Effort | Status |
|------|----------|------|----------|-----------------------------|---------|------------|--------|--------|
| 1 | P1 | [item] | [metric or finding] | [work item ID or explicit target] | [reason] | [high/med/low] | [S/M/L] | [open/blocked] |

## Selection for Next Iteration

- [Chosen item]
- [Why it wins the next slot]

## Review Checklist

- [ ] Each item cites evidence
- [ ] Tracker references are included
- [ ] Ordering is deterministic
diff --git a/docs/website/content/artifact-types/iterate/metric-definition.md b/docs/website/content/artifact-types/iterate/metric-definition.md index 44244004..f81ebe64 100644 --- a/docs/website/content/artifact-types/iterate/metric-definition.md +++ b/docs/website/content/artifact-types/iterate/metric-definition.md @@ -15,8 +15,9 @@ define exactly what is measured, how to collect it, what unit it uses, whether higher or lower is better, what tolerance applies, and how dashboards, ratchets, experiments, or monitoring should interpret it. -It is not a dashboard, alert policy, or improvement backlog item. Those artifacts -consume metric definitions. +For how this artifact relates to dashboards, security-metrics, and the +improvement backlog, see the "Metric Four-Way Slice" section of +`workflows/activities/06-iterate/README.md`. ## Example @@ -96,7 +97,7 @@ labels: RequiresNone EnablesNone InformsMetrics Dashboard
Monitoring Setup -Generation prompt
Show the full generation prompt
# Metric Definition Prompt

Create one reusable metric definition.

## Purpose

Metric Definition is the **single-measurement contract**. Its unique job is to
define exactly what is measured, how to collect it, what unit it uses, whether
higher or lower is better, what tolerance applies, and how dashboards,
ratchets, experiments, or monitoring should interpret it.

It is not a dashboard, alert policy, or improvement backlog item. Those artifacts
consume metric definitions.

## Reference Anchors

Use this local resource summary as grounding:

- `docs/resources/google-sre-monitoring-distributed-systems.md` grounds metric
  definitions as precise quantitative signals with clear interpretation.

## Focus
Define the metric as the authoritative source for ratchets, dashboards, experiments, and monitoring.

Keep the definition minimal: required fields are `name`, `description`, `unit`, `direction`, and `command`. Add `output_pattern`, `tolerance`, and `labels` only when needed.

The command must be deterministic, repeatable, and free of side effects or external service dependencies. Prefer `METRIC <name>=<value>` output unless an `output_pattern` is required.

## Boundary Test

| If you are writing... | Put it in... |
|---|---|
| One metric's unit, command, direction, tolerance, and labels | Metric Definition |
| A view comparing multiple metrics over time | Metrics Dashboard |
| A decision about what to improve next | Improvement Backlog |
| Production alerting or runbook behavior | Monitoring Setup / Runbook |

## Completion Criteria
- All required fields are populated.
- The command is deterministic and repeatable.
- The filename matches the `name` field.
+Generation prompt
Show the full generation prompt
# Metric Definition Prompt

Create one reusable metric definition.

## Purpose

Metric Definition is the **single-measurement contract**. Its unique job is to
define exactly what is measured, how to collect it, what unit it uses, whether
higher or lower is better, what tolerance applies, and how dashboards,
ratchets, experiments, or monitoring should interpret it.

For how this artifact relates to dashboards, security-metrics, and the
improvement backlog, see the "Metric Four-Way Slice" section of
`workflows/activities/06-iterate/README.md`.

## Reference Anchors

Use this local resource summary as grounding:

- `docs/resources/google-sre-monitoring-distributed-systems.md` grounds metric
  definitions as precise quantitative signals with clear interpretation.

## Focus
Define the metric as the authoritative source for ratchets, dashboards, experiments, and monitoring.

Keep the definition minimal: required fields are `name`, `description`, `unit`, `direction`, and `command`. Add `output_pattern`, `tolerance`, and `labels` only when needed.

The command must be deterministic, repeatable, and free of side effects or external service dependencies. Prefer `METRIC <name>=<value>` output unless an `output_pattern` is required.

## Boundary Test

| If you are writing... | Put it in... |
|---|---|
| One metric's unit, command, direction, tolerance, and labels | Metric Definition |
| A view comparing multiple metrics over time | Metrics Dashboard |
| A decision about what to improve next | Improvement Backlog |
| Production alerting or runbook behavior | Monitoring Setup / Runbook |

## Completion Criteria
- All required fields are populated.
- The command is deterministic and repeatable.
- The filename matches the `name` field.
Template
Show the template structure
---
ddx:
  id: METRIC-name
---

# Metric Definition: [NAME]

> Store at `docs/helix/06-iterate/metrics/[NAME].yaml`

```yaml
name: [kebab-case-identifier]
description: [What this metric measures]
unit: [seconds|bytes|percent|count|score|...]
direction: [lower|higher]
command: [repeatable shell command that actually runs and emits the value]
output_pattern: "[regex with capture group]"
baseline: [the value the command produced on a recorded run — measured, not asserted]
target: [the value that counts as success, in the same unit]
tolerance: [noise band, e.g. "5%" or "100ms"]
last_verified: [ISO-8601 date the command was last run and confirmed to emit this value]
interpretation: [How to read meaningful changes]
labels:
  [key]: [value]
```

**The measurement must be real.** `command` must have actually been run and
confirmed to emit the value before this metric is trusted; record that run date
in `last_verified`. A metric whose command was never run — an
asserted-but-unmeasured number — is a phantom claim (FEAT-016), not a metric.
`baseline` is what the command *produced*, never a target typed in by hand; for
a regression bench (a metric validating a methodology/skill change), `baseline`
is the bare-prompt reading and `target` is the value that earns the change its
keep.
diff --git a/docs/website/content/artifact-types/iterate/metrics-dashboard.md b/docs/website/content/artifact-types/iterate/metrics-dashboard.md index 4c35c2e8..d3f373f0 100644 --- a/docs/website/content/artifact-types/iterate/metrics-dashboard.md +++ b/docs/website/content/artifact-types/iterate/metrics-dashboard.md @@ -15,9 +15,9 @@ job is to compare current metric values against explicit baselines, interpret direction and tolerance, and produce a clear decision about improvement, regression, or noise. -It consumes Metric Definitions. It does not redefine metric formulas, command -semantics, or labels. It informs the Improvement Backlog but does not decide -implementation work by itself. +For how this artifact relates to metric definitions, security-metrics, and the +improvement backlog, see the "Metric Four-Way Slice" section of +`workflows/activities/06-iterate/README.md`. ## Example @@ -103,7 +103,7 @@ outside the 5% noise band, so the result counts as a meaningful improvement. EnablesNone InformsImprovement Backlog HELIX documentsdocs/helix/06-iterate/metrics-dashboard.md -Generation prompt
Show the full generation prompt
# Metrics Dashboard Generation Prompt

Document the iteration-level metrics summary used to judge whether the latest
changeset improved the system.

## Purpose

Metrics Dashboard is the **iteration-level measurement summary**. Its unique
job is to compare current metric values against explicit baselines, interpret
direction and tolerance, and produce a clear decision about improvement,
regression, or noise.

It consumes Metric Definitions. It does not redefine metric formulas, command
semantics, or labels. It informs the Improvement Backlog but does not decide
implementation work by itself.

## Reference Anchors

Use this local resource summary as grounding:

- `docs/resources/google-sre-monitoring-distributed-systems.md` grounds
  dashboard summaries as interpreted quantitative signals with clear sources.

## Focus
- Start from the canonical metric definitions in `docs/helix/06-iterate/metrics/`.
- Compare the current measurement against the previous baseline or committed floor.
- State whether the change improved, regressed, or stayed within noise.
- Include only the metrics needed to support the decision.
- Cite the source of each metric and the measurement command or report.
- Keep raw observability setup and implementation details out of this artifact.

## Boundary Test

| If you are writing... | Put it in... |
|---|---|
| Metric unit, command, tolerance, and labels | Metric Definition |
| Current-vs-baseline interpretation for one iteration | Metrics Dashboard |
| Prioritized follow-up work | Improvement Backlog |
| Alerting or dashboard implementation details | Monitoring Setup |

## Completion Criteria
- Every metric cited has a source definition and current value.
- The comparison baseline is explicit.
- The conclusion is actionable and easy to hand to the next iteration.
+Generation prompt
Show the full generation prompt
# Metrics Dashboard Generation Prompt

Document the iteration-level metrics summary used to judge whether the latest
changeset improved the system.

## Purpose

Metrics Dashboard is the **iteration-level measurement summary**. Its unique
job is to compare current metric values against explicit baselines, interpret
direction and tolerance, and produce a clear decision about improvement,
regression, or noise.

For how this artifact relates to metric definitions, security-metrics, and the
improvement backlog, see the "Metric Four-Way Slice" section of
`workflows/activities/06-iterate/README.md`.

## Reference Anchors

Use this local resource summary as grounding:

- `docs/resources/google-sre-monitoring-distributed-systems.md` grounds
  dashboard summaries as interpreted quantitative signals with clear sources.

## Focus
- Start from the canonical metric definitions in `docs/helix/06-iterate/metrics/`.
- Compare the current measurement against the previous baseline or committed floor.
- State whether the change improved, regressed, or stayed within noise.
- Include only the metrics needed to support the decision.
- Cite the source of each metric and the measurement command or report.
- Keep raw observability setup and implementation details out of this artifact.

## Boundary Test

| If you are writing... | Put it in... |
|---|---|
| Metric unit, command, tolerance, and labels | Metric Definition |
| Current-vs-baseline interpretation for one iteration | Metrics Dashboard |
| Prioritized follow-up work | Improvement Backlog |
| Alerting or dashboard implementation details | Monitoring Setup |

## Completion Criteria
- Every metric cited has a source definition and current value.
- The comparison baseline is explicit.
- The conclusion is actionable and easy to hand to the next iteration.
Template
Show the template structure
---
ddx:
  id: metrics-dashboard
---

# Metrics Dashboard: [iteration or release]

**Review Window**: [start date - end date]
**Baseline**: [previous iteration, ratchet floor, or benchmark]
**Status**: [draft | complete]

## Decision

[State whether the latest change improved the system and why.]

## Summary

[One concise paragraph that summarizes the result of the measured change.]

## Metrics Table

| Metric | Baseline | Current | Direction | Result | Source |
|--------|----------|---------|-----------|--------|--------|
| [name] | [value] | [value] | [higher/lower] | [pass/fail/noise] | [metric definition or report] |

## Interpretation Rules

- [How tolerance/noise is applied]
- [What creates a follow-up]

## Trend Notes

- [Trend or anomaly]
- [What changed relative to the baseline]

## Follow-Up

- [Tracker issue ID or next step]

## Review Checklist

- [ ] Baseline is explicit
- [ ] Each metric cites a source
- [ ] The summary states the decision implication
diff --git a/docs/website/content/artifact-types/iterate/security-metrics.md b/docs/website/content/artifact-types/iterate/security-metrics.md index 03b5a7ec..66f9dcb7 100644 --- a/docs/website/content/artifact-types/iterate/security-metrics.md +++ b/docs/website/content/artifact-types/iterate/security-metrics.md @@ -118,7 +118,7 @@ Each recommendation must be specific enough to create a tracker issue. EnablesNone InformsImprovement Backlog HELIX documentsdocs/helix/06-iterate/security-metrics.md -Generation prompt
Show the full generation prompt
# Security Metrics Prompt

Create a security metrics report for one iteration.

## Required Inputs
- Security monitoring and incident data for the iteration period
- Vulnerability scan results (SAST, DAST, dependency scans)
- Compliance audit findings, if applicable
- Previous security metrics report for trend comparison

## Reference Anchors

Use these local resource summaries as grounding:

- `docs/resources/nist-cybersecurity-measurement-guidance.md` grounds
  risk-based, trend-oriented security measurement.
- `docs/resources/owasp-asvs.md` grounds application-security control coverage.
- `docs/resources/google-sre-incident-management-guide.md` grounds incident
  response measurement and follow-up.

## Produced Output
- `docs/helix/06-iterate/security-metrics.md`

## Focus

Report on security posture across four areas: incident response, vulnerability
management, application security, and compliance. For each area, state the
current value, the target, and the trend. Do not repeat raw data in prose —
summarize what the numbers mean and what action they justify.
Separate production security metrics from product outcome metrics unless the
security signal directly changes operational risk.

Trend comparison against the previous period is required. If no prior report
exists, note the baseline and set targets for the next iteration.

Every recommendation must be specific enough to become a tracker issue. Vague
recommendations ("improve security posture") are not acceptable.

## Completion Criteria
- [ ] All four metric areas populated with current data
- [ ] Trend column populated for each metric (or baseline set if first report)
- [ ] At least one recommendation per area that is actionable as a tracker issue
- [ ] Root cause included for any critical or high-severity incidents
- [ ] Report covers the same iteration period as `metrics-dashboard.md`
- [ ] Raw scanner or incident output is summarized, not pasted wholesale

Use the template at `workflows/activities/06-iterate/artifacts/security-metrics/template.md`.
+Generation prompt
Show the full generation prompt
# Security Metrics Prompt

Create a security metrics report for one iteration.

For how this artifact relates to metric definitions, the metrics dashboard,
and the improvement backlog, see the "Metric Four-Way Slice" section of
`workflows/activities/06-iterate/README.md`.

## Required Inputs
- Security monitoring and incident data for the iteration period
- Vulnerability scan results (SAST, DAST, dependency scans)
- Compliance audit findings, if applicable
- Previous security metrics report for trend comparison

## Reference Anchors

Use these local resource summaries as grounding:

- `docs/resources/nist-cybersecurity-measurement-guidance.md` grounds
  risk-based, trend-oriented security measurement.
- `docs/resources/owasp-asvs.md` grounds application-security control coverage.
- `docs/resources/google-sre-incident-management-guide.md` grounds incident
  response measurement and follow-up.

## Produced Output
- `docs/helix/06-iterate/security-metrics.md`

## Focus

Report on security posture across four areas: incident response, vulnerability
management, application security, and compliance. For each area, state the
current value, the target, and the trend. Do not repeat raw data in prose —
summarize what the numbers mean and what action they justify.
Separate production security metrics from product outcome metrics unless the
security signal directly changes operational risk.

Trend comparison against the previous period is required. If no prior report
exists, note the baseline and set targets for the next iteration.

Every recommendation must be specific enough to become a tracker issue. Vague
recommendations ("improve security posture") are not acceptable.

## Completion Criteria
- [ ] All four metric areas populated with current data
- [ ] Trend column populated for each metric (or baseline set if first report)
- [ ] At least one recommendation per area that is actionable as a tracker issue
- [ ] Root cause included for any critical or high-severity incidents
- [ ] Report covers the same iteration period as `metrics-dashboard.md`
- [ ] Raw scanner or incident output is summarized, not pasted wholesale

Use the template at `workflows/activities/06-iterate/artifacts/security-metrics/template.md`.
Template
Show the template structure
---
ddx:
  id: security-metrics
---

# Security Metrics - [Iteration / Date Range]

## Incident Response

| Metric | Current | Target | Trend |
|--------|---------|--------|-------|
| Mean Time to Detect (MTTD) | | | |
| Mean Time to Respond (MTTR) | | | |
| Incidents resolved within SLA | | | |
| False-positive alert rate | | | |

**Incident Summary**

- Total incidents this period: [X]
- Critical (required immediate response): [X]
- Fully resolved: [X]

**Root Causes** (critical and high only)

| Root Cause | Count | Mitigation Status |
|------------|-------|-------------------|
| | | |

## Vulnerability Management

| Metric | Current | Target | Trend |
|--------|---------|--------|-------|
| Open critical vulnerabilities | | | |
| Open high vulnerabilities | | | |
| MTTR for critical vulns | | | |
| Patch compliance rate | | | |

## Application Security

| Metric | Current | Target | Trend |
|--------|---------|--------|-------|
| SAST findings (new this period) | | | |
| DAST findings (new this period) | | | |
| Dependency vulnerabilities (direct) | | | |
| Security review coverage | | | |

## Compliance Status

| Requirement | Status | Open Gaps | Target Resolution |
|-------------|--------|-----------|-------------------|
| | | | |

## Security Posture Trend

- **Overall risk level**: [Low / Medium / High] — Trend: [Improving / Stable / Declining]
- **Summary**: [One sentence on direction and primary driver]

## Recommendations

Each recommendation must be specific enough to create a tracker issue.

| Recommendation | Priority | Rationale | Expected Impact |
|----------------|----------|-----------|-----------------|
| | High / Med / Low | | |
diff --git a/docs/website/content/artifact-types/test/_index.md b/docs/website/content/artifact-types/test/_index.md index fc534f0e..fd5b381c 100644 --- a/docs/website/content/artifact-types/test/_index.md +++ b/docs/website/content/artifact-types/test/_index.md @@ -20,5 +20,5 @@ Define how we know it works. Plans, suites, and procedures that bind specs to im {{< card link="security-tests/" title="Security Tests" subtitle="Security verification matrix that maps material threats and controls to executable tests, evidence, and residual-risk decisions." >}} {{< card link="test-procedures/" title="Test Procedures" subtitle="Repeatable operator procedures for running planned tests, capturing evidence, and applying pass/fail rules consistently." >}} {{< card link="test-suites/" title="Test Suites" subtitle="Inventory of executable test suites with boundaries, coverage ownership, commands, runtime expectations, and evidence outputs." >}} - {{< card link="data-quality-expectations/" title="Data Quality Expectations" subtitle="Defines quality contracts as testable predicates on data shape, completeness, timeliness, and correctness before implementation begins. Data quality expectatio…" >}} + {{< card link="data-quality-expectations/" title="Data Quality Expectations" subtitle="data-quality-expectations" >}} {{< /cards >}} diff --git a/docs/website/content/artifact-types/test/data-quality-expectations.md b/docs/website/content/artifact-types/test/data-quality-expectations.md index 865071ba..4025ac49 100644 --- a/docs/website/content/artifact-types/test/data-quality-expectations.md +++ b/docs/website/content/artifact-types/test/data-quality-expectations.md @@ -10,7 +10,7 @@ generated: true ## Purpose -Data Quality Expectations is the **contract layer between Data PRD (what quality +Data Quality Expectations is the **contract layer between PRD (kind: data) (what quality is needed) and implementation** (how to verify it). Its unique job is to define measurable quality assertions as executable tests: data should not be released to consumers until these contracts are satisfied. @@ -29,7 +29,8 @@ written in a machine-readable form before code is written. ddx: id: example.data-quality-expectations.customer-360 depends_on: - - example.data-prd.customer-360 + # Previous: example.data-prd.customer-360 — dropped when data-prd + # collapsed into prd as kind: data variant (ADR-008). - example.data-architecture.customer-360 --- @@ -431,12 +432,9 @@ Release to BI (if GE-001, GE-002, GE-003 pass) - - - - - + +
ActivityTest — Define how we know it works. Plans, suites, and procedures that bind specs to implementation.
Default locationdocs/helix/03-test/data-quality-expectations.md
RequiresNone
EnablesNone
InformsTest Plan
Story Test Plan
Implementation Plan
Referenced byImplementation Plan
Runbook
Monitoring Setup
Generation prompt
Show the full generation prompt
# Data Quality Expectations Generation Prompt

Define the quality contracts that the data pipeline must satisfy, written as
testable predicates on data shape, completeness, accuracy, and freshness.

## Purpose

Data Quality Expectations is the **contract layer between Data PRD (what quality
is needed) and implementation** (how to verify it). Its unique job is to define
measurable quality assertions as executable tests: data should not be released
to consumers until these contracts are satisfied.

Unlike Test Plan (which is the overall test strategy) or Test Suites (which is
test code organization), Data Quality Expectations are the *quality commitments*
written in a machine-readable form before code is written.

## Reference Anchors

Use these local resource summaries as grounding:

- `docs/resources/databricks-sdp-expect.md` grounds Databricks Semantic Data
  Platform `EXPECT ... ON VIOLATION ...` syntax for inline quality assertions
  and contract-driven pipeline composition.
- `docs/resources/dbt-tests.md` grounds dbt's assertion language (tests,
  contracts, and constraints) for data quality as code.
- `docs/resources/great-expectations.md` grounds Great Expectations vocabulary,
  expectations, suites, and checkpoints for flexible, reusable quality checks.

## Focus

- For each requirement in the Data PRD, write at least one testable expectation.
- Group expectations by layer (Bronze, Silver, Gold) and validate at the
  appropriate layer.
- Use SDP `EXPECT` syntax for Databricks Streaming Tables, dbt tests for
  transformed tables, and Great Expectations for custom or complex checks.
- Name what happens when expectations fail: quarantine the data, alert,
  rollback, or skip dependent pipelines.
- Document freshness SLAs and how they are monitored (timestamp columns,
  row-count trends, lag detection).

## Role Boundary

Data Quality Expectations are not implementation code, not test infrastructure,
and not the data model. They are the *contract* that implementation must satisfy.

**Databricks Platform Substitution:** If you are adopting this on another data
platform, substitute as follows:

| Databricks Concept | Snowflake Equivalent | BigQuery Equivalent | On-Prem / Other |
|---|---|---|---|
| SDP `EXPECT ... ON VIOLATION ...` | Data Quality Checks + Task error handling | BigQuery Data Quality API + assertions | dbt tests, Great Expectations, SQL assertions |
| Streaming Tables with built-in `EXPECT` | Stream-triggered materialized views + constraints | Dataflow with Beam assertions | Flink-based pipelines with custom state |
| SDP Genie for test generation | dbt auto-generate tests from table samples | BigQuery Data Catalog insights | dbt, custom metadata scanning |
| Databricks Lakeview dashboards for monitoring | Snowflake Dashboards | Looker, Data Studio | Grafana, custom dashboards |

## Completion Criteria

- Every P0 requirement from Data PRD has at least one corresponding expectation.
- Expectations are grouped by layer (Bronze input validation, Silver
  transformation contracts, Gold business rule verification).
- Expectations are written in the platform's native syntax (SDP `EXPECT`,
  dbt test, Great Expectations).
- Each expectation names the failure mode: what happens if the check fails
  (quarantine, alert, skip downstream, rollback).
- Freshness SLAs are explicit: expected refresh interval and how lag is
  detected.
- Quality dashboards or monitoring strategy is sketched (where operators watch,
  what metrics matter, alert thresholds).
- Expectations are prioritized (which checks must pass for release, which are
  advisory).
Template
Show the template structure
---
ddx:
  id: data-quality-expectations
---

# Data Quality Expectations

## Overview and Scope

[Define the data product being tested, the medallion layers covered (Bronze, Silver, Gold), and the quality dimensions in scope (completeness, timeliness, accuracy, consistency, uniqueness). Reference the [[data-prd]] for quality requirements and [[data-architecture]] for the layer definitions. This document translates those requirements into executable EXPECT clauses.]

### Quality Dimensions

| Dimension | Definition | P0 Threshold | P1 Threshold | Enforcement |
|-----------|-----------|--------------|--------------|------------|
| Completeness | % of non-null values in key columns | ≥99% | ≥95% | Reject if P0 fails |
| Timeliness | Max age of data (now - max timestamp) | ≤1 hour | ≤4 hours | Alert if P0, skip if P1 |
| Accuracy | % of values matching source or business rules | ≥98% | ≥95% | Manual audit + alert |
| Uniqueness | No duplicate rows on primary key | 0 duplicates | ≤0.1% | Reject if P0 fails |
| Consistency | Cross-layer contracts (sums reconcile, cardinality stable) | ±0.01% | ±0.1% | Reject if P0 fails |

### Test Framework and Tooling

- **Framework**: [SDP EXPECT clauses / dbt tests / Great Expectations / SQL constraints]
- **Execution**: [Databricks Workflows, dbt Cloud, scheduled notebooks]
- **Alerting**: [Slack + email to data-eng on-call]
- **Remediation**: [Manual data fix, pipeline rollback, quarantine until reviewed]

### Testing Philosophy

[Exhaustive testing on all rows (default) or sampling? Document the rationale. Sampling is OK for high-volume tables and computationally expensive checks, but must include a confidence interval and margin of error.]

---

## Bronze Layer Expectations

### Raw Data Validation

Bronze tables land source data without transformation. Expectations here catch ingestion failures and schema drift early.

### Schema and Structure

```sql
-- Customers Bronze: all source columns present, no truncation
EXPECT TABLE raw_customers (
  customer_id NOT NULL,
  email NOT NULL,
  phone,
  created_at TIMESTAMP NOT NULL,
  updated_at TIMESTAMP NOT NULL,
  _source_system STRING NOT NULL,
  _ingest_timestamp TIMESTAMP NOT NULL
);

-- Data types match source
EXPECT TABLE raw_customers (
  CHECK (customer_id IS INT),
  CHECK (email IS STRING),
  CHECK (created_at IS TIMESTAMP)
);
```

**Severity**: Blocking — fail ingestion if schema mismatch.
**Action on Failure**: Stop ingest pipeline; alert data-eng; manual review before retry.

### Completeness (Null Check)

```sql
-- Critical columns must never be null
EXPECT TABLE raw_customers
  CHECK (customer_id IS NOT NULL)
  CHECK (email IS NOT NULL);

-- If any of these are null, it's a quality failure (not a schema error)
EXPECT TABLE raw_customers (
  completeness_check AS (
    SUM(CASE WHEN customer_id IS NULL THEN 1 ELSE 0 END) 
      / COUNT(*) < 0.01  -- P0: <1% nulls on customer_id
  )
);
```

**Severity**: Blocking (P0) — reject records with null customer_id.
**Threshold**: <1% nulls on critical columns.
**Action on Failure**: Quarantine batch; alert; wait for manual approval.

### Freshness (Timeliness)

```sql
-- Ingestion must complete within 15 minutes of source commit
EXPECT TABLE raw_customers (
  freshness_check AS (
    MAX(_ingest_timestamp) >= CURRENT_TIMESTAMP() - INTERVAL 15 MINUTES
  )
);
```

**Severity**: Warning (P1) — alert if >15 min old; move to Silver anyway but flag.
**Action on Failure**: Alert to on-call; do not advance to Silver if >30 min.

### No Truncation

```sql
-- Source columns must be preserved at full width (no truncation to shorter types)
EXPECT TABLE raw_customers (
  CHECK (LENGTH(email) <= 255),  -- assume source is VARCHAR(255)
  CHECK (LENGTH(phone) <= 20)
);
```

**Severity**: Warning — audit only.
**Action on Failure**: Log; manual spot-check.

---

## Silver Layer Expectations

### Validation and Transformation

Silver tables apply business logic: deduplication, null handling, type coercion, referential integrity. Expectations here enforce "clean" data for consumption.

### Uniqueness (Deduplication)

```sql
-- Each customer appears exactly once (deduplicated by latest timestamp)
EXPECT TABLE customers_validated (
  PRIMARY KEY (customer_id),
  UNIQUE (customer_id)
);

EXPECT TABLE customers_validated (
  uniqueness_check AS (
    COUNT(*) = COUNT(DISTINCT customer_id)
  )
);
```

**Severity**: Blocking — reject if duplicates.
**Threshold**: 0 duplicates on PK.
**Action on Failure**: Fail pipeline; manual dedup review; rerun.

### Completeness (Post-Transform)

```sql
-- After validation, critical columns must be NOT NULL
EXPECT TABLE customers_validated (
  CHECK (customer_id IS NOT NULL),
  CHECK (email IS NOT NULL)
);

EXPECT TABLE customers_validated (
  CHECK (
    (SUM(CASE WHEN email IS NULL THEN 1 ELSE 0 END) / COUNT(*)) < 0.01
  )  -- <1% nulls on email
);
```

**Severity**: Blocking (P0).
**Threshold**: 0% nulls on customer_id; <1% nulls on email.
**Action on Failure**: Reject batch; alert.

### Data Quality and Normalization

```sql
-- Email addresses are normalized and valid
EXPECT TABLE customers_validated (
  CHECK (
    email LIKE '%@%.%' AND email NOT LIKE ' %' AND email NOT LIKE '% '
  )
);

-- Phone numbers (if present) are valid format
EXPECT TABLE customers_validated (
  CHECK (
    phone IS NULL OR phone REGEXP '^[0-9\-\+\(\) ]+$'
  )
);

-- Created_at is before or equal to updated_at
EXPECT TABLE customers_validated (
  CHECK (created_at <= updated_at)
);
```

**Severity**: Warning (P1) — log invalid records but don't block.
**Action on Failure**: Alert + audit; filter invalid records for Gold.

### Referential Integrity

```sql
-- If this table is child of a parent, all foreign keys must exist
-- (e.g., customer_segment.customer_id must exist in customers_validated)
EXPECT TABLE customers_validated (
  referential_integrity_check AS (
    NOT EXISTS (
      SELECT 1 FROM customer_segment cs
      WHERE NOT EXISTS (
        SELECT 1 FROM customers_validated cv
        WHERE cv.customer_id = cs.customer_id
      )
    )
  )
);
```

**Severity**: Blocking (P0) for critical relationships.
**Action on Failure**: Reject segment batch; alert; manual review.

### Timeliness (Staleness)

```sql
-- Silver must be refreshed at least daily (no more than 24 hours stale)
EXPECT TABLE customers_validated (
  freshness_check AS (
    MAX(_modified_at) >= CURRENT_TIMESTAMP() - INTERVAL 1 DAY
  )
);
```

**Severity**: Warning (P1).
**SLA**: Alert if >24 hours old.
**Action on Failure**: Alert to on-call; check if pipeline is hung.

### Row Count Reconciliation

```sql
-- Silver row count must be close to Bronze (within dedup margin)
-- Account for legitimate filtering (e.g., test records, deleted accounts)
EXPECT TABLE customers_validated (
  reconciliation_check AS (
    (SELECT COUNT(*) FROM customers_validated)
    BETWEEN
      (SELECT COUNT(*) FROM raw_customers) * 0.90  -- allow 10% loss for dedup
      AND (SELECT COUNT(*) FROM raw_customers) * 1.01  -- allow 1% gain for corrections
  )
);
```

**Severity**: Blocking (P0).
**Tolerance**: ±10% (dedup and corrections).
**Action on Failure**: Reject; alert; manual audit.

---

## Gold Layer Expectations

### Business-Ready Consumption

Gold tables are optimized for consumer queries and serve as the source of truth for analytics, ML, and BI. Expectations here enforce freshness, consistency, and accuracy for downstream consumers.

### Completeness and Current State

```sql
-- Customer 360 Gold table is current within SLA
EXPECT TABLE customer_360 (
  freshness_check AS (
    MAX(_modified_at) >= CURRENT_TIMESTAMP() - INTERVAL 1 HOUR
  )
);

-- All customer_ids in customer_360 exist in Silver (no orphans)
EXPECT TABLE customer_360 (
  orphan_check AS (
    NOT EXISTS (
      SELECT 1 FROM customer_360 c360
      WHERE NOT EXISTS (
        SELECT 1 FROM customers_validated cv
        WHERE cv.customer_id = c360.customer_id
      )
    )
  )
);
```

**Severity**: Blocking (P0).
**SLA**: <1 hour stale.
**Action on Failure**: Reject; roll back to prior snapshot; alert on-call.

### Aggregate Accuracy and Reconciliation

```sql
-- Daily sales summary must reconcile with Silver within $0.01 per order
EXPECT TABLE daily_sales_summary (
  aggregate_reconciliation AS (
    SELECT SUM(amount) FROM daily_sales_summary
    IS WITHIN 0.01 OF
    SELECT SUM(order_amount) FROM orders_silver
  )
);

-- Row counts match between daily and historical gold
EXPECT TABLE daily_sales_summary (
  cardinality_check AS (
    COUNT(DISTINCT DATE(order_date)) = COUNT(DISTINCT DATE(order_date)) OVER (PARTITION BY MONTH(order_date))
  )
);
```

**Severity**: Blocking (P0) — finances depend on this.
**Tolerance**: ±$0.01 per order (handle floating-point rounding).
**Action on Failure**: Reject; re-aggregate; alert finance team.

### Consumer Guarantees (SLA)

```sql
-- Query performance: p95 query latency must be <5 seconds for customer_360
-- (Monitored via Databricks query history, not an EXPECT clause, but documented here)

-- Availability: customer_360 must be queryable during business hours (8am-8pm UTC)
EXPECT TABLE customer_360 (
  availability_check AS (
    TABLE customer_360 IS NOT EMPTY  -- simple check; better: monitor cluster uptime
  )
);
```

**Severity**: Warning (P1) — operational SLA.
**Action on Failure**: Page on-call; investigate cluster/query issues.

### Consistency Across Related Gold Tables

```sql
-- If customer_360 and daily_sales_summary both exist:
-- Sum of amounts per customer in daily_sales_summary 
-- should match lifetime_revenue in customer_360
EXPECT (
  SELECT SUM(amount) FROM daily_sales_summary WHERE customer_id = X
  EQUALS
  SELECT lifetime_revenue FROM customer_360 WHERE customer_id = X
);
```

**Severity**: Blocking (P0) for critical aggregates.
**Action on Failure**: Reject; investigate transformation logic; recompute.

### Business Logic Validation

```sql
-- Revenue should never be negative
EXPECT TABLE daily_sales_summary (
  CHECK (amount >= 0)
);

-- Order dates should be within reasonable bounds (not in future, not before company founded)
EXPECT TABLE daily_sales_summary (
  CHECK (order_date BETWEEN '2015-01-01' AND CURRENT_DATE)
);
```

**Severity**: Blocking (P0).
**Action on Failure**: Reject; alert; audit source data.

---

## Cross-Layer Contracts

### Data Lineage and Consistency

Expectations that span multiple layers ensure data doesn't transform incorrectly as it flows Bronze → Silver → Gold.

### Layer-to-Layer Validation

| Contract | Assertion | If Violated | Severity |
|----------|-----------|----------|----------|
| Bronze → Silver row count | Silver rows ≥ 90% of Bronze rows (allow dedup) | Investigate dedup logic | Blocking |
| Silver → Gold row count | Gold unique customers = Silver unique customers | Reject Gold; re-aggregate | Blocking |
| Bronze → Gold data types | Gold numbers can be summed without loss of precision | Audit source → Gold transformation | Blocking |

### Cross-Table Contracts (Fact-Dimension)

```sql
-- All orders.customer_id must exist in customer_360
EXPECT TABLE orders_gold (
  fk_customer_check AS (
    NOT EXISTS (
      SELECT 1 FROM orders_gold og
      WHERE NOT EXISTS (
        SELECT 1 FROM customer_360 c360
        WHERE c360.customer_id = og.customer_id
      )
    )
  )
);

-- Sum of order amounts per customer must match customer lifetime_revenue
EXPECT (
  SELECT SUM(amount) FROM orders_gold GROUP BY customer_id
  EQUALS
  SELECT lifetime_revenue FROM customer_360
);
```

**Severity**: Blocking (P0).
**Action on Failure**: Quarantine orders; alert; manual reconciliation.

---

## Failure Handling and SLA

### Alert and Escalation Policy

| Expectation | Severity | Detection SLA | Escalation | Action |
|-----------|----------|--------------|-----------|--------|
| [Bronze completeness] | Blocking | <5 min | Page on-call | Pause ingest; investigate |
| [Silver uniqueness] | Blocking | <10 min | Slack + email | Stop pipeline; manual dedup review |
| [Gold freshness] | Blocking | <15 min | Page + email | Investigate scheduler; rerun |
| [Aggregate reconciliation] | Blocking | <30 min | Finance + on-call | Recompute; audit |
| [Query latency] | Warning | <1 min (logged) | Email on-call | Investigate cluster; optimize queries |

### Failure Recovery

**On Blocking Failure**:
1. Stop the pipeline (no downstream advancement)
2. Alert on-call data engineer immediately
3. Log detailed failure context (which rows failed, why)
4. Quarantine the batch in a `_quarantine_` location for manual review
5. Do not retry automatically; require manual approval + fix before re-processing

**On Warning Failure**:
1. Log the failure
2. Send alert to Slack (not pager)
3. Continue pipeline; flag data as low-confidence
4. Schedule manual audit within 24 hours

### SLA Target

- **Detection**: <5 min for Bronze, <15 min for Silver/Gold
- **Mean Time to Recovery (MTTR)**: <30 min for blocking failures
- **False Positive Rate**: <1% (tune thresholds to reduce alert fatigue)

---

## Review Checklist

Use this checklist during review to validate that the data quality expectations are complete and testable:

- [ ] **Overview and Scope** clearly states which medallion layers and quality dimensions are covered
- [ ] **Bronze Layer Expectations** validate schema, completeness, freshness, and ingestion integrity
- [ ] **Silver Layer Expectations** enforce deduplication, nullability, normalization, and referential integrity
- [ ] **Gold Layer Expectations** guarantee freshness, consistency, and accuracy for consumer queries
- [ ] **Expectations are written in executable form**: SDP EXPECT, dbt test, or SQL constraint (not prose)
- [ ] **Each expectation traces back** to a quality requirement in [[data-prd]] (reference by name)
- [ ] **Failure modes are explicit**: What happens when an expectation fails (reject, alert, quarantine)?
- [ ] **SLA per layer** is documented: freshness target, detection time, recovery time
- [ ] **Sampling vs exhaustive** is clear: All rows tested, or sample with confidence interval documented?
- [ ] **Cross-layer data contracts** ensure consistency across layers (sums reconcile, cardinality stable, no orphans)
- [ ] **Alert routing and escalation policy** is defined: Who gets paged? When does on-call respond?
- [ ] **No `[TBD]`, `[TODO]`, or `[NEEDS CLARIFICATION]` markers remain**
- [ ] **At least one expectation per quality dimension** (completeness, timeliness, accuracy, consistency, uniqueness)
- [ ] **P0 requirements have multiple expectations** (layered checks: Bronze schema, Silver uniqueness, Gold freshness)
- [ ] **Terminology aligns with Databricks SDP** (EXPECT clauses, UC, medallion layers, VIOLATION rules)
Generation prompt
Show the full generation prompt
# Data Quality Expectations Generation Prompt

Define the quality contracts that the data pipeline must satisfy, written as
testable predicates on data shape, completeness, accuracy, and freshness.

## Purpose

Data Quality Expectations is the **contract layer between PRD (kind: data) (what quality
is needed) and implementation** (how to verify it). Its unique job is to define
measurable quality assertions as executable tests: data should not be released
to consumers until these contracts are satisfied.

Unlike Test Plan (which is the overall test strategy) or Test Suites (which is
test code organization), Data Quality Expectations are the *quality commitments*
written in a machine-readable form before code is written.

## Reference Anchors

Use these local resource summaries as grounding:

- `docs/resources/databricks-sdp-expect.md` grounds Databricks Semantic Data
  Platform `EXPECT ... ON VIOLATION ...` syntax for inline quality assertions
  and contract-driven pipeline composition.
- `docs/resources/dbt-tests.md` grounds dbt's assertion language (tests,
  contracts, and constraints) for data quality as code.
- `docs/resources/great-expectations.md` grounds Great Expectations vocabulary,
  expectations, suites, and checkpoints for flexible, reusable quality checks.

## Focus

- For each requirement in the PRD (kind: data), write at least one testable expectation.
- Group expectations by layer (Bronze, Silver, Gold) and validate at the
  appropriate layer.
- Use SDP `EXPECT` syntax for Databricks Streaming Tables, dbt tests for
  transformed tables, and Great Expectations for custom or complex checks.
- Name what happens when expectations fail: quarantine the data, alert,
  rollback, or skip dependent pipelines.
- Document freshness SLAs and how they are monitored (timestamp columns,
  row-count trends, lag detection).

## Role Boundary

Data Quality Expectations are not implementation code, not test infrastructure,
and not the data model. They are the *contract* that implementation must satisfy.

**Databricks Platform Substitution:** If you are adopting this on another data
platform, substitute as follows:

| Databricks Concept | Snowflake Equivalent | BigQuery Equivalent | On-Prem / Other |
|---|---|---|---|
| SDP `EXPECT ... ON VIOLATION ...` | Data Quality Checks + Task error handling | BigQuery Data Quality API + assertions | dbt tests, Great Expectations, SQL assertions |
| Streaming Tables with built-in `EXPECT` | Stream-triggered materialized views + constraints | Dataflow with Beam assertions | Flink-based pipelines with custom state |
| SDP Genie for test generation | dbt auto-generate tests from table samples | BigQuery Data Catalog insights | dbt, custom metadata scanning |
| Databricks Lakeview dashboards for monitoring | Snowflake Dashboards | Looker, Data Studio | Grafana, custom dashboards |

## Completion Criteria

- Every P0 requirement from PRD (kind: data) has at least one corresponding expectation.
- Expectations are grouped by layer (Bronze input validation, Silver
  transformation contracts, Gold business rule verification).
- Expectations are written in the platform's native syntax (SDP `EXPECT`,
  dbt test, Great Expectations).
- Each expectation names the failure mode: what happens if the check fails
  (quarantine, alert, skip downstream, rollback).
- Freshness SLAs are explicit: expected refresh interval and how lag is
  detected.
- Quality dashboards or monitoring strategy is sketched (where operators watch,
  what metrics matter, alert thresholds).
- Expectations are prioritized (which checks must pass for release, which are
  advisory).
Template
Show the template structure
---
ddx:
  id: data-quality-expectations
---

# Data Quality Expectations

## Overview and Scope

[Define the data product being tested, the medallion layers covered (Bronze, Silver, Gold), and the quality dimensions in scope (completeness, timeliness, accuracy, consistency, uniqueness). Reference the [[prd]] (kind: data) for quality requirements and [[data-architecture]] for the layer definitions. This document translates those requirements into executable EXPECT clauses.]

### Quality Dimensions

| Dimension | Definition | P0 Threshold | P1 Threshold | Enforcement |
|-----------|-----------|--------------|--------------|------------|
| Completeness | % of non-null values in key columns | ≥99% | ≥95% | Reject if P0 fails |
| Timeliness | Max age of data (now - max timestamp) | ≤1 hour | ≤4 hours | Alert if P0, skip if P1 |
| Accuracy | % of values matching source or business rules | ≥98% | ≥95% | Manual audit + alert |
| Uniqueness | No duplicate rows on primary key | 0 duplicates | ≤0.1% | Reject if P0 fails |
| Consistency | Cross-layer contracts (sums reconcile, cardinality stable) | ±0.01% | ±0.1% | Reject if P0 fails |

### Test Framework and Tooling

- **Framework**: [SDP EXPECT clauses / dbt tests / Great Expectations / SQL constraints]
- **Execution**: [Databricks Workflows, dbt Cloud, scheduled notebooks]
- **Alerting**: [Slack + email to data-eng on-call]
- **Remediation**: [Manual data fix, pipeline rollback, quarantine until reviewed]

### Testing Philosophy

[Exhaustive testing on all rows (default) or sampling? Document the rationale. Sampling is OK for high-volume tables and computationally expensive checks, but must include a confidence interval and margin of error.]

---

## Bronze Layer Expectations

### Raw Data Validation

Bronze tables land source data without transformation. Expectations here catch ingestion failures and schema drift early.

### Schema and Structure

```sql
-- Customers Bronze: all source columns present, no truncation
EXPECT TABLE raw_customers (
  customer_id NOT NULL,
  email NOT NULL,
  phone,
  created_at TIMESTAMP NOT NULL,
  updated_at TIMESTAMP NOT NULL,
  _source_system STRING NOT NULL,
  _ingest_timestamp TIMESTAMP NOT NULL
);

-- Data types match source
EXPECT TABLE raw_customers (
  CHECK (customer_id IS INT),
  CHECK (email IS STRING),
  CHECK (created_at IS TIMESTAMP)
);
```

**Severity**: Blocking — fail ingestion if schema mismatch.
**Action on Failure**: Stop ingest pipeline; alert data-eng; manual review before retry.

### Completeness (Null Check)

```sql
-- Critical columns must never be null
EXPECT TABLE raw_customers
  CHECK (customer_id IS NOT NULL)
  CHECK (email IS NOT NULL);

-- If any of these are null, it's a quality failure (not a schema error)
EXPECT TABLE raw_customers (
  completeness_check AS (
    SUM(CASE WHEN customer_id IS NULL THEN 1 ELSE 0 END) 
      / COUNT(*) < 0.01  -- P0: <1% nulls on customer_id
  )
);
```

**Severity**: Blocking (P0) — reject records with null customer_id.
**Threshold**: <1% nulls on critical columns.
**Action on Failure**: Quarantine batch; alert; wait for manual approval.

### Freshness (Timeliness)

```sql
-- Ingestion must complete within 15 minutes of source commit
EXPECT TABLE raw_customers (
  freshness_check AS (
    MAX(_ingest_timestamp) >= CURRENT_TIMESTAMP() - INTERVAL 15 MINUTES
  )
);
```

**Severity**: Warning (P1) — alert if >15 min old; move to Silver anyway but flag.
**Action on Failure**: Alert to on-call; do not advance to Silver if >30 min.

### No Truncation

```sql
-- Source columns must be preserved at full width (no truncation to shorter types)
EXPECT TABLE raw_customers (
  CHECK (LENGTH(email) <= 255),  -- assume source is VARCHAR(255)
  CHECK (LENGTH(phone) <= 20)
);
```

**Severity**: Warning — audit only.
**Action on Failure**: Log; manual spot-check.

---

## Silver Layer Expectations

### Validation and Transformation

Silver tables apply business logic: deduplication, null handling, type coercion, referential integrity. Expectations here enforce "clean" data for consumption.

### Uniqueness (Deduplication)

```sql
-- Each customer appears exactly once (deduplicated by latest timestamp)
EXPECT TABLE customers_validated (
  PRIMARY KEY (customer_id),
  UNIQUE (customer_id)
);

EXPECT TABLE customers_validated (
  uniqueness_check AS (
    COUNT(*) = COUNT(DISTINCT customer_id)
  )
);
```

**Severity**: Blocking — reject if duplicates.
**Threshold**: 0 duplicates on PK.
**Action on Failure**: Fail pipeline; manual dedup review; rerun.

### Completeness (Post-Transform)

```sql
-- After validation, critical columns must be NOT NULL
EXPECT TABLE customers_validated (
  CHECK (customer_id IS NOT NULL),
  CHECK (email IS NOT NULL)
);

EXPECT TABLE customers_validated (
  CHECK (
    (SUM(CASE WHEN email IS NULL THEN 1 ELSE 0 END) / COUNT(*)) < 0.01
  )  -- <1% nulls on email
);
```

**Severity**: Blocking (P0).
**Threshold**: 0% nulls on customer_id; <1% nulls on email.
**Action on Failure**: Reject batch; alert.

### Data Quality and Normalization

```sql
-- Email addresses are normalized and valid
EXPECT TABLE customers_validated (
  CHECK (
    email LIKE '%@%.%' AND email NOT LIKE ' %' AND email NOT LIKE '% '
  )
);

-- Phone numbers (if present) are valid format
EXPECT TABLE customers_validated (
  CHECK (
    phone IS NULL OR phone REGEXP '^[0-9\-\+\(\) ]+$'
  )
);

-- Created_at is before or equal to updated_at
EXPECT TABLE customers_validated (
  CHECK (created_at <= updated_at)
);
```

**Severity**: Warning (P1) — log invalid records but don't block.
**Action on Failure**: Alert + audit; filter invalid records for Gold.

### Referential Integrity

```sql
-- If this table is child of a parent, all foreign keys must exist
-- (e.g., customer_segment.customer_id must exist in customers_validated)
EXPECT TABLE customers_validated (
  referential_integrity_check AS (
    NOT EXISTS (
      SELECT 1 FROM customer_segment cs
      WHERE NOT EXISTS (
        SELECT 1 FROM customers_validated cv
        WHERE cv.customer_id = cs.customer_id
      )
    )
  )
);
```

**Severity**: Blocking (P0) for critical relationships.
**Action on Failure**: Reject segment batch; alert; manual review.

### Timeliness (Staleness)

```sql
-- Silver must be refreshed at least daily (no more than 24 hours stale)
EXPECT TABLE customers_validated (
  freshness_check AS (
    MAX(_modified_at) >= CURRENT_TIMESTAMP() - INTERVAL 1 DAY
  )
);
```

**Severity**: Warning (P1).
**SLA**: Alert if >24 hours old.
**Action on Failure**: Alert to on-call; check if pipeline is hung.

### Row Count Reconciliation

```sql
-- Silver row count must be close to Bronze (within dedup margin)
-- Account for legitimate filtering (e.g., test records, deleted accounts)
EXPECT TABLE customers_validated (
  reconciliation_check AS (
    (SELECT COUNT(*) FROM customers_validated)
    BETWEEN
      (SELECT COUNT(*) FROM raw_customers) * 0.90  -- allow 10% loss for dedup
      AND (SELECT COUNT(*) FROM raw_customers) * 1.01  -- allow 1% gain for corrections
  )
);
```

**Severity**: Blocking (P0).
**Tolerance**: ±10% (dedup and corrections).
**Action on Failure**: Reject; alert; manual audit.

---

## Gold Layer Expectations

### Business-Ready Consumption

Gold tables are optimized for consumer queries and serve as the source of truth for analytics, ML, and BI. Expectations here enforce freshness, consistency, and accuracy for downstream consumers.

### Completeness and Current State

```sql
-- Customer 360 Gold table is current within SLA
EXPECT TABLE customer_360 (
  freshness_check AS (
    MAX(_modified_at) >= CURRENT_TIMESTAMP() - INTERVAL 1 HOUR
  )
);

-- All customer_ids in customer_360 exist in Silver (no orphans)
EXPECT TABLE customer_360 (
  orphan_check AS (
    NOT EXISTS (
      SELECT 1 FROM customer_360 c360
      WHERE NOT EXISTS (
        SELECT 1 FROM customers_validated cv
        WHERE cv.customer_id = c360.customer_id
      )
    )
  )
);
```

**Severity**: Blocking (P0).
**SLA**: <1 hour stale.
**Action on Failure**: Reject; roll back to prior snapshot; alert on-call.

### Aggregate Accuracy and Reconciliation

```sql
-- Daily sales summary must reconcile with Silver within $0.01 per order
EXPECT TABLE daily_sales_summary (
  aggregate_reconciliation AS (
    SELECT SUM(amount) FROM daily_sales_summary
    IS WITHIN 0.01 OF
    SELECT SUM(order_amount) FROM orders_silver
  )
);

-- Row counts match between daily and historical gold
EXPECT TABLE daily_sales_summary (
  cardinality_check AS (
    COUNT(DISTINCT DATE(order_date)) = COUNT(DISTINCT DATE(order_date)) OVER (PARTITION BY MONTH(order_date))
  )
);
```

**Severity**: Blocking (P0) — finances depend on this.
**Tolerance**: ±$0.01 per order (handle floating-point rounding).
**Action on Failure**: Reject; re-aggregate; alert finance team.

### Consumer Guarantees (SLA)

```sql
-- Query performance: p95 query latency must be <5 seconds for customer_360
-- (Monitored via Databricks query history, not an EXPECT clause, but documented here)

-- Availability: customer_360 must be queryable during business hours (8am-8pm UTC)
EXPECT TABLE customer_360 (
  availability_check AS (
    TABLE customer_360 IS NOT EMPTY  -- simple check; better: monitor cluster uptime
  )
);
```

**Severity**: Warning (P1) — operational SLA.
**Action on Failure**: Page on-call; investigate cluster/query issues.

### Consistency Across Related Gold Tables

```sql
-- If customer_360 and daily_sales_summary both exist:
-- Sum of amounts per customer in daily_sales_summary 
-- should match lifetime_revenue in customer_360
EXPECT (
  SELECT SUM(amount) FROM daily_sales_summary WHERE customer_id = X
  EQUALS
  SELECT lifetime_revenue FROM customer_360 WHERE customer_id = X
);
```

**Severity**: Blocking (P0) for critical aggregates.
**Action on Failure**: Reject; investigate transformation logic; recompute.

### Business Logic Validation

```sql
-- Revenue should never be negative
EXPECT TABLE daily_sales_summary (
  CHECK (amount >= 0)
);

-- Order dates should be within reasonable bounds (not in future, not before company founded)
EXPECT TABLE daily_sales_summary (
  CHECK (order_date BETWEEN '2015-01-01' AND CURRENT_DATE)
);
```

**Severity**: Blocking (P0).
**Action on Failure**: Reject; alert; audit source data.

---

## Cross-Layer Contracts

### Data Lineage and Consistency

Expectations that span multiple layers ensure data doesn't transform incorrectly as it flows Bronze → Silver → Gold.

### Layer-to-Layer Validation

| Contract | Assertion | If Violated | Severity |
|----------|-----------|----------|----------|
| Bronze → Silver row count | Silver rows ≥ 90% of Bronze rows (allow dedup) | Investigate dedup logic | Blocking |
| Silver → Gold row count | Gold unique customers = Silver unique customers | Reject Gold; re-aggregate | Blocking |
| Bronze → Gold data types | Gold numbers can be summed without loss of precision | Audit source → Gold transformation | Blocking |

### Cross-Table Contracts (Fact-Dimension)

```sql
-- All orders.customer_id must exist in customer_360
EXPECT TABLE orders_gold (
  fk_customer_check AS (
    NOT EXISTS (
      SELECT 1 FROM orders_gold og
      WHERE NOT EXISTS (
        SELECT 1 FROM customer_360 c360
        WHERE c360.customer_id = og.customer_id
      )
    )
  )
);

-- Sum of order amounts per customer must match customer lifetime_revenue
EXPECT (
  SELECT SUM(amount) FROM orders_gold GROUP BY customer_id
  EQUALS
  SELECT lifetime_revenue FROM customer_360
);
```

**Severity**: Blocking (P0).
**Action on Failure**: Quarantine orders; alert; manual reconciliation.

---

## Failure Handling and SLA

### Alert and Escalation Policy

| Expectation | Severity | Detection SLA | Escalation | Action |
|-----------|----------|--------------|-----------|--------|
| [Bronze completeness] | Blocking | <5 min | Page on-call | Pause ingest; investigate |
| [Silver uniqueness] | Blocking | <10 min | Slack + email | Stop pipeline; manual dedup review |
| [Gold freshness] | Blocking | <15 min | Page + email | Investigate scheduler; rerun |
| [Aggregate reconciliation] | Blocking | <30 min | Finance + on-call | Recompute; audit |
| [Query latency] | Warning | <1 min (logged) | Email on-call | Investigate cluster; optimize queries |

### Failure Recovery

**On Blocking Failure**:
1. Stop the pipeline (no downstream advancement)
2. Alert on-call data engineer immediately
3. Log detailed failure context (which rows failed, why)
4. Quarantine the batch in a `_quarantine_` location for manual review
5. Do not retry automatically; require manual approval + fix before re-processing

**On Warning Failure**:
1. Log the failure
2. Send alert to Slack (not pager)
3. Continue pipeline; flag data as low-confidence
4. Schedule manual audit within 24 hours

### SLA Target

- **Detection**: <5 min for Bronze, <15 min for Silver/Gold
- **Mean Time to Recovery (MTTR)**: <30 min for blocking failures
- **False Positive Rate**: <1% (tune thresholds to reduce alert fatigue)

---

## Review Checklist

Use this checklist during review to validate that the data quality expectations are complete and testable:

- [ ] **Overview and Scope** clearly states which medallion layers and quality dimensions are covered
- [ ] **Bronze Layer Expectations** validate schema, completeness, freshness, and ingestion integrity
- [ ] **Silver Layer Expectations** enforce deduplication, nullability, normalization, and referential integrity
- [ ] **Gold Layer Expectations** guarantee freshness, consistency, and accuracy for consumer queries
- [ ] **Expectations are written in executable form**: SDP EXPECT, dbt test, or SQL constraint (not prose)
- [ ] **Each expectation traces back** to a quality requirement in [[prd]] (kind: data) (reference by name)
- [ ] **Failure modes are explicit**: What happens when an expectation fails (reject, alert, quarantine)?
- [ ] **SLA per layer** is documented: freshness target, detection time, recovery time
- [ ] **Sampling vs exhaustive** is clear: All rows tested, or sample with confidence interval documented?
- [ ] **Cross-layer data contracts** ensure consistency across layers (sums reconcile, cardinality stable, no orphans)
- [ ] **Alert routing and escalation policy** is defined: Who gets paged? When does on-call respond?
- [ ] **No `[TBD]`, `[TODO]`, or `[NEEDS CLARIFICATION]` markers remain**
- [ ] **At least one expectation per quality dimension** (completeness, timeliness, accuracy, consistency, uniqueness)
- [ ] **P0 requirements have multiple expectations** (layered checks: Bronze schema, Silver uniqueness, Gold freshness)
- [ ] **Terminology aligns with Databricks SDP** (EXPECT clauses, UC, medallion layers, VIOLATION rules)
diff --git a/docs/website/content/artifact-types/test/story-test-plan.md b/docs/website/content/artifact-types/test/story-test-plan.md index 4d5933cb..2f1cfa76 100644 --- a/docs/website/content/artifact-types/test/story-test-plan.md +++ b/docs/website/content/artifact-types/test/story-test-plan.md @@ -10,13 +10,10 @@ generated: true ## Purpose -Story Test Plan is the **story-level executable verification handoff**. Its -unique job is to turn one user story's acceptance criteria and technical design -into concrete failing tests, fixtures, commands, and setup before Build starts. - -It inherits the project Test Plan. It does not redefine test strategy, coverage -targets, or feature-wide risk. It owns the exact evidence needed for one story -slice. +The story test plan translates one bounded technical design into executable +verification intent before implementation starts. It maps story acceptance +criteria to concrete failing tests, names the required setup and data, and +gives Build a precise handoff for one story-sized slice. ## Example @@ -172,7 +169,7 @@ pnpm test:e2e -- upload-csv EnablesNone InformsTest Suites
Implementation Plan Referenced byTechnical Design
Implementation Plan -Generation prompt
Show the full generation prompt
# Story Test Plan Generation Prompt

Create the canonical story-scoped test plan for one bounded story slice. This
artifact exists because the project-wide `test-plan.md` does not replace the
need for per-story acceptance-to-test traceability.

## Purpose

Story Test Plan is the **story-level executable verification handoff**. Its
unique job is to turn one user story's acceptance criteria and technical design
into concrete failing tests, fixtures, commands, and setup before Build starts.

It inherits the project Test Plan. It does not redefine test strategy, coverage
targets, or feature-wide risk. It owns the exact evidence needed for one story
slice.

## Reference Anchors

Use these local resource summaries as grounding:

- `docs/resources/cucumber-executable-specifications.md` grounds acceptance
  criteria as observable executable examples.
- `docs/resources/google-test-sizes.md` grounds story test levels by scope,
  dependency, and execution cost.

## Storage Location

`docs/helix/03-test/test-plans/STP-{id}-{name}.md`

## What to Include

- the governing `[[US-XXX-*]]` and `[[TD-XXX-*]]` references
- a tight scope statement plus explicit out-of-scope boundaries
- a matrix mapping each active acceptance criterion to concrete failing tests,
  keyed by the story's **stable `US-<n>-AC<m>` ID** (this story-level matrix is
  the AC↔test traceability surface; the project test plan aggregates strategy
  and allocates layers but does not duplicate these rows — FEAT-008 FR-6). Each
  row names the **behavior/assertion the test makes** (the observable outcome it
  checks), not just a test name — a named test with no named assertion does not
  show the criterion is *exercised*. Each row also names the **covering test AND
  records that the test cites the AC ID** in the canonical, parseable syntax
  `@covers US-<n>-AC<m>` — a test that exercises and passes but does not cite the
  AC ID is `UNCITED_COVERAGE` (not covered for traceability; fix = add the
  citation, not a new test), distinct from `UNTESTED`. Citation is an additional
  gate on top of exercise+pass+satisfy, never a replacement
- executable proof details: test file paths, commands, or named test cases
- setup, fixtures, seed data, mocks, and environment assumptions
- edge cases and error scenarios that the story must prove before build begins
- build handoff notes that help implementation sequence the work

## Minimum Quality Bar

- Stay story-scoped. Do not drift into feature-wide strategy or generic testing doctrine.
- Name runnable evidence, not just test categories.
- Prefer one compact mapping table over repeated prose.
- If an acceptance criterion is not being covered now, say why explicitly.

## Boundary Test

| If you are writing... | Put it in... |
|---|---|
| Project-wide test levels, coverage, and CI gates | Test Plan |
| One story's concrete tests, fixtures, commands, and setup | Story Test Plan |
| Product behavior or acceptance criteria | User Story / Feature Specification |
| Implementation file changes | Technical Design / Implementation Plan |

Use template at `workflows/activities/03-test/artifacts/story-test-plan/template.md`.
+Generation prompt
Show the full generation prompt
# Story Test Plan Generation Prompt

Scope: one story's acceptance-criteria-to-test traceability — concrete failing tests, fixtures, commands, and setup for a single bounded slice.

Related:
- [Test Plan](../test-plan/prompt.md) — project-wide strategy this STP inherits
- [Test Suites](../test-suites/prompt.md) — where these story tests live under `tests/`
- [Test Procedures](../test-procedures/prompt.md) — how tests get written and run

## Reference Anchors

Use these local resource summaries as grounding:

- `docs/resources/cucumber-executable-specifications.md` grounds acceptance
  criteria as observable executable examples.
- `docs/resources/google-test-sizes.md` grounds story test levels by scope,
  dependency, and execution cost.

## Storage Location

`docs/helix/03-test/test-plans/STP-{id}-{name}.md`

## What to Include

- the governing `[[US-XXX-*]]` and `[[TD-XXX-*]]` references
- a tight scope statement plus explicit out-of-scope boundaries
- a matrix mapping each active acceptance criterion to concrete failing tests,
  keyed by the story's **stable `US-<n>-AC<m>` ID** (this story-level matrix is
  the AC↔test traceability surface; the project test plan aggregates strategy
  and allocates layers but does not duplicate these rows — FEAT-008 FR-6). Each
  row names the **behavior/assertion the test makes** (the observable outcome it
  checks), not just a test name — a named test with no named assertion does not
  show the criterion is *exercised*. Each row also names the **covering test AND
  records that the test cites the AC ID** in the canonical, parseable syntax
  `@covers US-<n>-AC<m>` — a test that exercises and passes but does not cite the
  AC ID is `UNCITED_COVERAGE` (not covered for traceability; fix = add the
  citation, not a new test), distinct from `UNTESTED`. Citation is an additional
  gate on top of exercise+pass+satisfy, never a replacement
- executable proof details: test file paths, commands, or named test cases
- setup, fixtures, seed data, mocks, and environment assumptions
- edge cases and error scenarios that the story must prove before build begins
- build handoff notes that help implementation sequence the work

## Minimum Quality Bar

- Stay story-scoped. Do not drift into feature-wide strategy or generic testing doctrine.
- Name runnable evidence, not just test categories.
- Prefer one compact mapping table over repeated prose.
- If an acceptance criterion is not being covered now, say why explicitly.

## Boundary Test

| If you are writing... | Put it in... |
|---|---|
| Project-wide test levels, coverage, and CI gates | Test Plan |
| One story's concrete tests, fixtures, commands, and setup | Story Test Plan |
| Product behavior or acceptance criteria | User Story / Feature Specification |
| Implementation file changes | Technical Design / Implementation Plan |

Use template at `workflows/activities/03-test/artifacts/story-test-plan/template.md`.
Template
Show the template structure
---
ddx:
  id: STP-XXX
---

# Story Test Plan: STP-XXX-[story-name]

## Story Reference

**User Story**: [[US-XXX-[story-name]]]
**Technical Design**: [[TD-XXX-[story-name]]]
**Related Solution Design**: [[SD-XXX-[feature-name]]] or N/A
**Project Test Plan**: [[test-plan]]

## Scope and Objective

**Goal**: [What this story must prove before build starts]
**Blocking Gate**: [Command or suite that must pass for this story]

**In Scope**
- [Bounded behavior this TP governs]

**Out of Scope**
- [Adjacent behavior intentionally left to another TP, feature, or future slice]

## Acceptance Criteria Test Mapping

This matrix is the **story-level** AC↔test traceability surface. Key each row on
the story's **stable AC ID** (`US-<n>-AC<m>`) so a specific criterion maps to a
specific failing test. This story test plan owns this matrix; the project-level
test plan aggregates strategy and allocates layers — it does **not** duplicate
these rows (FEAT-008 FR-6).

Each row must name the **behavior the test asserts** — the specific observable
outcome it checks — not merely a test name. A row that lists a test name with no
named assertion does not prove the criterion is *exercised*; reconcile-alignment
classifies such a criterion `UNTESTED` (or `ASSERTED_UNBACKED` if the named test
does not exist), not covered.

Each row must also name the **covering test AND record that the test cites the
AC ID** in the canonical, parseable syntax `@covers US-<n>-AC<m>` (the structured
tag in the test body, name, or doc comment). The citation makes AC→test
traceability machine-checkable. A test that exercises and passes but does **not**
cite the AC ID is classified `UNCITED_COVERAGE` (not covered for traceability;
the fix is to add the citation, not a new test) — distinct from `UNTESTED`. The
citation is an *additional* gate on top of exercise+pass+satisfy, never a
replacement. The canonical, parseable citation syntax is `@covers US-<n>-AC<m>`
with numeric stable IDs (e.g. `@covers US-001-AC1`); `US-XXX` below is a
placeholder for the numeric story id — replace `XXX` with the real number.

| AC ID | Acceptance Criterion (Given/When/Then) | Failing Test(s) to Create or Run | Asserted Behavior (what the test verifies) | AC-ID Citation (`@covers`) | Test Level | File or Command | Setup / Data | Notes |
|-------|----------------------------------------|----------------------------------|--------------------------------------------|----------------------------|------------|-----------------|--------------|-------|
| US-001-AC1 | [Given/When/Then criterion] | `[test_name]` | [the concrete outcome the test asserts — e.g. "response is 200 with body {id}"] | `@covers US-001-AC1` | Unit / Integration / Contract / E2E | `tests/...` or `bash ...` | [Fixture, seed, mock] | [Edge case or sequencing note] |

## Executable Proof

### Primary Commands

```bash
[command that proves this TP]
```

### Planned Test Files

- `tests/...`
- `tests/...`

### Coverage Focus

- P0: [Must-pass behavior]
- P1: [Important secondary behavior]

## Data and Setup

| Need | Required For | Source / Strategy |
|------|--------------|-------------------|
| [Fixture / seed / mock / env var] | [Tests] | [Where it comes from] |

## Edge Cases and Failure Modes

- [Boundary value or empty-state handling]
- [Validation failure or invalid input]
- [Dependency failure, timeout, or permission edge]

## Build Handoff

**Implementation Order**
1. [What should be implemented first to turn the first red test green]
2. [What follows once the core path passes]

**Constraints**
- [Constraint inherited from requirements, design, concern, or contract]

**Done When**
- [ ] Every in-scope acceptance criterion has passing evidence
- [ ] Named commands or test files exist and run
- [ ] Out-of-scope coverage remains explicitly deferred rather than silently skipped
- [ ] The story can fail red before implementation and pass green after implementation

## Review Checklist

- [ ] References the governing story and technical design
- [ ] Every active acceptance criterion maps to concrete failing tests, keyed by its stable `US-<n>-AC<m>` ID
- [ ] Every AC row names the behavior/assertion the test makes, not just a test name
- [ ] Every AC row names the covering test AND records its `@covers US-<n>-AC<m>` citation (canonical AC-ID syntax)
- [ ] File paths, commands, or test identifiers are specific enough to execute
- [ ] Setup, fixtures, mocks, and seed data are explicit
- [ ] Edge cases cover real story risks rather than generic boilerplate
- [ ] Scope remains bounded to one story slice
- [ ] Build handoff gives implementation a usable sequence
diff --git a/docs/website/content/artifact-types/test/test-plan.md b/docs/website/content/artifact-types/test/test-plan.md index 0777b489..8af4d74d 100644 --- a/docs/website/content/artifact-types/test/test-plan.md +++ b/docs/website/content/artifact-types/test/test-plan.md @@ -10,14 +10,11 @@ generated: true ## Purpose -The Test Plan is the **project-level verification strategy**. Its unique job is -to define test levels, coverage targets, critical paths, data strategy, -infrastructure, sequencing, risks, and build handoff commands before -implementation starts. - -It does not contain every story-specific test case. Story Test Plans own the -exact executable checks for one story. The Test Plan owns the portfolio: what -must be covered, where confidence should come from, and how CI enforces it. +The project-level test plan defines the testing strategy for the full +project: test levels and scope, framework choices, coverage targets, +critical paths, test data strategy, infrastructure requirements, and +sequencing. It drives failing tests before implementation begins and +provides traceability from requirements to test execution. ## Example @@ -154,7 +151,7 @@ coverage minimums hold; no raw financial fixture values appear in logs. InformsImplementation Plan Referenced byPRD HELIX documentsdocs/helix/03-test/test-plans/TP-014-helix-workflow-coverage.md -Generation prompt
Show the full generation prompt
# Test Plan Generation Prompt

Create the project-level test plan for the Test activity. Keep it concise, but include the minimum structure needed to drive failing tests before implementation.

## Purpose

The Test Plan is the **project-level verification strategy**. Its unique job is
to define test levels, coverage targets, critical paths, data strategy,
infrastructure, sequencing, risks, and build handoff commands before
implementation starts.

It does not contain every story-specific test case. Story Test Plans own the
exact executable checks for one story. The Test Plan owns the portfolio: what
must be covered, where confidence should come from, and how CI enforces it.

## Reference Anchors

Use these local resource summaries as grounding:

- `docs/resources/google-test-sizes.md` grounds test levels by scope,
  isolation, dependencies, and CI enforcement.
- `docs/resources/fowler-practical-test-pyramid.md` grounds balanced coverage
  across fast focused tests and fewer broad end-to-end tests.

## Storage Location

`docs/helix/03-test/test-plan.md`

## What to Include

- test levels and scope
- framework choices only where they matter
- coverage targets and critical paths
- acceptance-criteria **layer allocation** — allocate each in-scope
  `US-<n>-AC<m>` criterion class to a primary test layer and confirm every P0 is
  allocated. **Aggregate** the story test plans; do **not** duplicate their
  per-AC matrix (the STP owns that, keyed by stable AC ID — FEAT-008 FR-6).
- test data strategy
- sequencing, dependencies, and infrastructure needs
- risks that can block test execution

## Keep In Mind

- tests are the executable specification
- every test should trace to a requirement or story
- coverage targets should be risk-based and enforced, not decorative
- do not add generic testing doctrine that the template already implies

## Boundary Test

| If you are writing... | Put it in... |
|---|---|
| Overall test levels, coverage targets, data strategy, CI gates | Test Plan |
| One story's concrete test cases and fixtures | Story Test Plan |
| Feature behavior or acceptance criteria | Feature Specification or User Story |
| Implementation sequencing for code changes | Implementation Plan |

Use template at `workflows/activities/03-test/artifacts/test-plan/template.md`.
+Generation prompt
Show the full generation prompt
# Test Plan Generation Prompt

Scope: project-level verification strategy — test levels, coverage targets, critical paths, data strategy, infrastructure, sequencing, risks, and build handoff commands.

Related:
- [Story Test Plan](../story-test-plan/prompt.md) — per-story AC↔test traceability
- [Test Suites](../test-suites/prompt.md) — suite inventory and boundaries under `tests/`
- [Test Procedures](../test-procedures/prompt.md) — runner procedures, commands, evidence capture

## Reference Anchors

Use these local resource summaries as grounding:

- `docs/resources/google-test-sizes.md` grounds test levels by scope,
  isolation, dependencies, and CI enforcement.
- `docs/resources/fowler-practical-test-pyramid.md` grounds balanced coverage
  across fast focused tests and fewer broad end-to-end tests.

## Storage Location

`docs/helix/03-test/test-plan.md`

## What to Include

- test levels and scope
- framework choices only where they matter
- coverage targets and critical paths
- acceptance-criteria **layer allocation** — allocate each in-scope
  `US-<n>-AC<m>` criterion class to a primary test layer and confirm every P0 is
  allocated. **Aggregate** the story test plans; do **not** duplicate their
  per-AC matrix (the STP owns that, keyed by stable AC ID — FEAT-008 FR-6).
- test data strategy
- sequencing, dependencies, and infrastructure needs
- risks that can block test execution

## Keep In Mind

- tests are the executable specification
- every test should trace to a requirement or story
- coverage targets should be risk-based and enforced, not decorative
- do not add generic testing doctrine that the template already implies

## Boundary Test

| If you are writing... | Put it in... |
|---|---|
| Overall test levels, coverage targets, data strategy, CI gates | Test Plan |
| One story's concrete test cases and fixtures | Story Test Plan |
| Feature behavior or acceptance criteria | Feature Specification or User Story |
| Implementation sequencing for code changes | Implementation Plan |

Use template at `workflows/activities/03-test/artifacts/test-plan/template.md`.
Template
Show the template structure
---
ddx:
  id: TP-XXX
---

# Test Plan

## Testing Strategy

**Goals**: [Primary objective] | [Quality gates]
**Out of Scope**: [Excluded areas]
**Traceability Source**: [PRD / FEAT / US artifacts that drive the plan]

### Test Levels

| Level | Coverage Target | Priority |
|-------|-----------------|----------|
| Contract | [Target and scope] | P0/P1 |
| Integration | [Target and scope] | P0/P1 |
| Unit | [Target and scope] | P0/P1 |
| E2E | [Target and scope] | P0/P1 |

### Frameworks

| Type | Framework | Reason |
|------|-----------|--------|
| Contract | [Framework] | [Why] |
| Integration | [Framework] | [Why] |
| Unit | [Framework] | [Why] |
| E2E | [Framework] | [Why] |

## Test Data

| Type | Strategy |
|------|----------|
| Fixtures | [Static data approach] |
| Factories | [Dynamic generation] |
| Mocks | [External service mocking] |

## Coverage Requirements

| Metric | Target | Minimum | Enforcement |
|--------|--------|---------|-------------|
| Line | 80% | 70% | CI blocks |
| Critical | 100% | 100% | Required |

### Critical Paths (P0)

1. [Auth flow]
2. [Core transaction]
3. [Data persistence]
4. [Error handling]

### Secondary Paths (P1-P2)

- P1: [Secondary features] | P2: [Edge cases, rare scenarios]

## Acceptance Criteria Layer Allocation

This project test plan **aggregates** strategy across stories. It does **not**
restate the per-criterion AC↔test matrix — that lives in each story test plan
(STP), keyed by stable `US-<n>-AC<m>` IDs (FEAT-008 FR-6). Here, allocate
criterion *classes* to test layers and confirm every P0 criterion is allocated:

| AC class / source | Story Test Plan(s) | Primary Layer | Why this layer |
|-------------------|--------------------|---------------|----------------|
| [e.g. upload/validation criteria] | [[STP-XXX]] | Integration | [boundary + persistence] |
| [e.g. visible reviewer flow] | [[STP-XXX]] | E2E | [user-observable outcome] |

**Allocation rule**: no P0 acceptance criterion is left unallocated — every
`US-<n>-AC<m>` from an in-scope story maps to exactly one primary layer here and
to concrete tests in its STP. The STP owns the per-AC rows; this plan owns the
layer allocation.

## Implementation Order
1. [What must be written first and why]
2. [What follows]
3. [What can wait]

## Infrastructure

| Requirement | Specification |
|-------------|---------------|
| CI Tool | [Tool, version] |
| Test DB | [Type, seeding, cleanup] |
| Services | [Required services] |

## Risks

| Risk | Impact | Mitigation |
|------|--------|------------|
| Flaky tests | High | Retry logic, isolation |
| Slow execution | Med | Parallelize |

**Known Gaps**: [Limitations and accepted risks]

## Build Handoff

**Commands**: `[test command]` | `[coverage command]`
**Priority**: [Recommended order]

**Blocking Gate**: [What must pass before implementation is considered done]

## Review Checklist

Use this checklist when reviewing a test plan:

- [ ] Test levels cover contract, integration, unit, and E2E with coverage targets
- [ ] Framework choices are justified and consistent with project concerns
- [ ] Critical paths (P0) are identified and have 100% coverage requirements
- [ ] Test data strategy covers fixtures, factories, and mocks
- [ ] Coverage requirements have both targets and minimums with enforcement rules
- [ ] Implementation order is justified — what must be tested first and why
- [ ] Infrastructure requirements are specific (tool versions, service deps)
- [ ] Risks include flaky test mitigation and slow execution strategies
- [ ] Known gaps are documented with accepted risk rationale
- [ ] Build handoff commands are concrete and runnable
- [ ] Test plan traces back to acceptance criteria from governing feature specs
- [ ] No untested P0 requirement — every P0 acceptance criterion has a test
- [ ] Acceptance criteria are allocated to test layers by AC class without
      duplicating the per-AC `US-<n>-AC<m>` matrix owned by the story test plans
diff --git a/docs/website/content/artifact-types/test/test-procedures.md b/docs/website/content/artifact-types/test/test-procedures.md index 6868d4a7..e3ea893d 100644 --- a/docs/website/content/artifact-types/test/test-procedures.md +++ b/docs/website/content/artifact-types/test/test-procedures.md @@ -158,7 +158,7 @@ contract, integration, unit, or security suite fails. RequiresNone EnablesNone InformsTest Suites -Generation prompt
Show the full generation prompt
# Test Procedures Generation Prompt

Create concise procedures for writing, running, and maintaining tests. Focus on the steps an implementer needs for this project, not general testing advice.

## Reference Anchors

Use these local resource summaries as grounding:

- `docs/resources/google-test-sizes.md` grounds different execution
  expectations for small, medium, and large tests.
- `docs/resources/fowler-practical-test-pyramid.md` grounds the balance between
  fast focused tests and fewer broad confidence tests.
- `docs/resources/cucumber-executable-specifications.md` grounds procedures for
  acceptance examples when behavior needs shared language.

## Storage Location

`docs/helix/03-test/test-procedures.md`

## Include

- per-test-type writing rules only where they differ
- local and CI execution commands
- validation and quality checks
- the common failure modes worth documenting
- evidence capture and pass/fail rules

Use template at `workflows/activities/03-test/artifacts/test-procedures/template.md`.
+Generation prompt
Show the full generation prompt
# Test Procedures Generation Prompt

Scope: procedures for writing, running, and maintaining tests — per-type writing rules, execution commands, validation checks, and evidence capture.

Related:
- [Test Plan](../test-plan/prompt.md) — project verification strategy these procedures execute
- [Story Test Plan](../story-test-plan/prompt.md) — per-story tests these procedures apply to
- [Test Suites](../test-suites/prompt.md) — suite layout these procedures operate on

## Reference Anchors

Use these local resource summaries as grounding:

- `docs/resources/google-test-sizes.md` grounds different execution
  expectations for small, medium, and large tests.
- `docs/resources/fowler-practical-test-pyramid.md` grounds the balance between
  fast focused tests and fewer broad confidence tests.
- `docs/resources/cucumber-executable-specifications.md` grounds procedures for
  acceptance examples when behavior needs shared language.

## Storage Location

`docs/helix/03-test/test-procedures.md`

## Include

- per-test-type writing rules only where they differ
- local and CI execution commands
- validation and quality checks
- the common failure modes worth documenting
- evidence capture and pass/fail rules

Use template at `workflows/activities/03-test/artifacts/test-procedures/template.md`.
Template
Show the template structure
---
ddx:
  id: test-procedures
---

# Test Procedures

## Scope

- Tests covered: [suite names, story IDs, risks, or acceptance criteria]
- Operators: [who runs or reviews the procedures]
- Out of scope: [tests or environments not covered here]

## Prerequisites

- [ ] Test framework configured
- [ ] CI pipeline ready
- [ ] Test data and mocks prepared

## Procedures

### Contract Tests

1. Review the API or interface contract.
2. Add one file per endpoint or command.
3. Cover success, validation, auth, and error paths.
4. Verify the test fails before implementation exists.

### Integration Tests

1. Review dependencies and data flow.
2. Use the smallest realistic mix of real and mocked services.
3. Cover coordination and failure handling.

### Unit Tests

1. Isolate business rules and edge cases.
2. Keep each test focused on one behavior.
3. Confirm the test fails for the right reason.

## Execution

### Local
```bash
npm test
npm test -- --coverage
npm test -- --watch
```

### CI

Run the same suite on push and PRs. Block merges on failure.

## Evidence Capture

| Procedure | Evidence | Location |
|-----------|----------|----------|
| [Procedure] | [Command output, CI run, screenshot, review note] | [Path/link] |

## Pass/Fail Rules

| Procedure | Pass | Fail |
|-----------|------|------|
| [Procedure] | [Observable passing condition] | [Blocking failure condition] |

## Quality Checklist

- [ ] Test names describe behavior
- [ ] Tests are independent and deterministic
- [ ] Assertions are specific
- [ ] Managed fixtures or factories are used

## Troubleshooting

| Problem | Likely Cause | Fix |
|---------|--------------|-----|
| Passing too early | Implementation or weak assertions | Recheck expectations |
| Poor failure messages | Test is too broad | Split the test |
| Flaky results | Time, state, or external deps | Remove dependency or isolate state |

## Handoff

- [ ] Required tests are written and failing
- [ ] CI is configured
- [ ] Test docs are complete
- [ ] Evidence and pass/fail rules are recorded
diff --git a/docs/website/content/artifact-types/test/test-suites.md b/docs/website/content/artifact-types/test/test-suites.md index c33e192e..4541506f 100644 --- a/docs/website/content/artifact-types/test/test-suites.md +++ b/docs/website/content/artifact-types/test/test-suites.md @@ -146,7 +146,7 @@ npx playwright test tests/e2e/reviewer-smoke.spec.ts RequiresNone EnablesNone InformsDeployment Checklist -Generation prompt
Show the full generation prompt
# Test Suites Generation Prompt

Create the test suite layout for the Red activity. Keep it concise and project-specific: define the suite boundaries, the minimum behavior each suite must cover, and any shared data conventions needed to make the tests executable.

## Reference Anchors

Use these local resource summaries as grounding:

- `docs/resources/fowler-practical-test-pyramid.md` grounds suite balance across
  unit, integration, and end-to-end coverage.
- `docs/resources/google-test-sizes.md` grounds runtime and dependency
  expectations for suite grouping.
- `docs/resources/openapi-specification.md` grounds API contract suites where
  interface contracts exist.

## Storage Location

`tests/` at the project root

## Include

- contract, integration, unit, and E2E boundaries
- the behaviors each suite owns
- required fixtures, factories, or mocks
- any coverage target that matters for this stack
- suite ownership, execution commands, and evidence outputs

## Keep Out

- generic TDD teaching text
- oversized code examples
- repeated explanations of why tests come first

Use template at `workflows/activities/03-test/artifacts/test-suites/template.md`.
+Generation prompt
Show the full generation prompt
# Test Suites Generation Prompt

Scope: suite layout under `tests/` — boundaries (contract/integration/unit/E2E), behaviors each suite owns, shared fixtures, and execution commands.

Related:
- [Test Plan](../test-plan/prompt.md) — project verification strategy these suites realize
- [Story Test Plan](../story-test-plan/prompt.md) — per-story tests that land in these suites
- [Test Procedures](../test-procedures/prompt.md) — writing, running, and maintaining the tests in these suites

## Reference Anchors

Use these local resource summaries as grounding:

- `docs/resources/fowler-practical-test-pyramid.md` grounds suite balance across
  unit, integration, and end-to-end coverage.
- `docs/resources/google-test-sizes.md` grounds runtime and dependency
  expectations for suite grouping.
- `docs/resources/openapi-specification.md` grounds API contract suites where
  interface contracts exist.

## Storage Location

`tests/` at the project root

## Include

- contract, integration, unit, and E2E boundaries
- the behaviors each suite owns
- required fixtures, factories, or mocks
- any coverage target that matters for this stack
- suite ownership, execution commands, and evidence outputs

## Keep Out

- generic TDD teaching text
- oversized code examples
- repeated explanations of why tests come first

Use template at `workflows/activities/03-test/artifacts/test-suites/template.md`.
Template
Show the template structure
---
ddx:
  id: test-suites
---

# Test Suite Structure

**Project**: [Project Name]
**Test Framework**: [Framework]

## Suite Inventory

| Suite | Path | Scope | Runtime | Required |
|-------|------|-------|---------|----------|
| Unit | `tests/unit/` | [Business rules and pure functions] | [Target] | Yes |
| Integration | `tests/integration/` | [Component and persistence flows] | [Target] | Yes |
| Contract | `tests/contract/` | [API/interface contract] | [Target] | Yes |
| E2E | `tests/e2e/` | [Critical user journeys] | [Target] | [Yes/No] |
| Security | `tests/security/` | [Threat/control checks] | [Target] | [Yes/No] |

## Coverage Mapping

## Contract Tests

| Requirement / Contract | Suite | Test File | Coverage |
|------------------------|-------|-----------|----------|
| [Contract item] | Contract | [file] | [Success, validation, auth, error] |

## Integration Tests

| Flow | Suite | Test File | Coverage |
|------|-------|-----------|----------|
| [Flow] | Integration | [file] | [Coordination, persistence, failure] |

## Unit Tests

| Rule / Module | Suite | Test File | Coverage |
|---------------|-------|-----------|----------|
| [Rule] | Unit | [file] | [Cases] |

## Security Tests

| Threat / Control | Suite | Test File | Coverage |
|------------------|-------|-----------|----------|
| [Threat] | Security | [file] | [Control behavior] |

## Test Data

| Asset | Purpose |
|-------|---------|
| Fixtures | [Canonical valid, invalid, and edge payloads] |
| Factories | [Generated test objects] |
| Mocks | [External services or time/network controls] |

## Execution Commands

```bash
[unit command]
[integration command]
[contract command]
[security command]
```

## Ownership

| Suite | Owner | Review Trigger |
|-------|-------|----------------|
| [Suite] | [Owner] | [When this suite changes] |

## Evidence

| Suite | Evidence Output | Required in CI |
|-------|-----------------|----------------|
| [Suite] | [Path/link] | [Yes/No] |

## Readiness
- [ ] Suite boundaries are defined
- [ ] Shared test data assets are identified
- [ ] All planned suites begin in RED
- [ ] Commands, owners, and evidence outputs are recorded
diff --git a/docs/website/content/artifacts/_index.md b/docs/website/content/artifacts/_index.md index 8eb3f12c..d993b841 100644 --- a/docs/website/content/artifacts/_index.md +++ b/docs/website/content/artifacts/_index.md @@ -49,7 +49,7 @@ _Auto-generated from `helix/` by `scripts/publish-artifacts.py`._ - [Rename "authority order" to "authority hierarchy"](/artifacts/plan-2026-05-29-rename-authority-order/) - [Artifact-type and concern audit — gaps and remediation plan](/artifacts/plan-2026-05-30-artifact-types-and-concerns-audit/) - [Security Architecture — DDx Agent Execution Surface](/artifacts/security-architecture/) -- [adr](/artifacts/adr/) _(2 items)_ +- [adr](/artifacts/adr/) _(13 items)_ - [contracts](/artifacts/contracts/) _(5 items)_ - [decisions](/artifacts/decisions/) _(2 items)_ - [hero image concepts](/artifacts/hero-image-concepts/) _(1 item)_ diff --git a/docs/website/content/artifacts/adr/ADR-004-dependencies-encoding.md b/docs/website/content/artifacts/adr/ADR-004-dependencies-encoding.md new file mode 100644 index 00000000..cb137bfc --- /dev/null +++ b/docs/website/content/artifacts/adr/ADR-004-dependencies-encoding.md @@ -0,0 +1,123 @@ +--- +title: "ADR-004: Artifact Dependencies Are Encoded in `meta.yml.relationships` Only (No Separate `dependencies.yaml`)" +slug: ADR-004-dependencies-encoding +weight: 190 +activity: "Design" +source: "02-design/adr/ADR-004-dependencies-encoding.md" +generated: true +collection: adr +--- + +> **Source identity** (from `02-design/adr/ADR-004-dependencies-encoding.md`): + +```yaml +ddx: + id: ADR-004 + depends_on: + - helix.prd +``` + +# ADR-004: Artifact Dependencies Are Encoded in `meta.yml.relationships` Only (No Separate `dependencies.yaml`) + +| Date | Status | Deciders | Related | Confidence | +|------|--------|----------|---------|------------| +| 2026-05-30 | Accepted | HELIX maintainers | artifact-schema, frame action, audit plan 2026-05-30 | High | + +## Context + +| Aspect | Description | +|--------|-------------| +| Problem | The artifact-type rubric historically expected each artifact-type directory to ship a separate `dependencies.yaml` file describing inputs, outputs, and validation rules. In practice, no such file exists for any artifact type, while the same information is already encoded in `meta.yml` under `relationships` (and adjacent `validation.*` sections). The audit (`docs/helix/02-design/plan-2026-05-30-artifact-types-and-concerns-audit.md`) found `dependencies.yaml` absent for 48/48 artifact types. | +| Current State | `meta.yml.relationships` already names `depends_on` and `informs` per artifact type. `meta.yml.validation` already carries `required_sections`, `quality_checks`, and friends. The phantom `dependencies.yaml` reference survives only in two prose locations: `workflows/actions/frame.md` (validation-gate step) and `workflows/activities/01-frame/README.md` (artifact-directory description). No consumer script actually opens a `dependencies.yaml` file: `scripts/helix_align_check.py` parses the spec stack from markdown (PRD, FEAT, US, ADR) and does not touch `dependencies.yaml`; `workflows/actions/reconcile-alignment.md` references "dependencies" only as a conceptual notion. | +| Requirements | The catalog must have a single source of truth for artifact-type dependencies. The rubric must stop asking for a file the catalog does not produce. Any consumer that currently mentions `dependencies.yaml` must be repointed at `meta.yml` so the rubric and the workflow agree. | + +## Decision + +We ratify **`meta.yml.relationships`** as the canonical encoding of artifact-type +dependencies, and we drop `dependencies.yaml` from the artifact-type rubric. +There is no second file. There is no generated projection. + +- Type-level dependencies (`depends_on`, `informs`, peer relations) live in + `meta.yml` under `relationships`. The structured shape is already defined in + `workflows/artifact-schema.md` under "Recommended fields". +- Type-level validation (`required_sections`, `quality_checks`, + `pattern_checks`, `automated_checks`) lives in `meta.yml` under `validation`. +- Instance-level dependencies (one artifact instance depending on another) + continue to live in instance frontmatter under `ddx.depends_on`, exactly as + `workflows/artifact-schema.md` already specifies. ADR-004 does not change + that. + +**Key Points**: one source of truth (`meta.yml`) | no `dependencies.yaml` +generation step | consumer prose realigned to `meta.yml` + +## Consumer-side compatibility + +We choose **option (a): update consumers to read `meta.yml` directly** over +option (b): generate a transient `dependencies.yaml` projection at build/check +time. Rationale: + +- `scripts/helix_align_check.py` does not currently read `dependencies.yaml` at + all. It parses markdown (`prd.md`, `FEAT-*.md`, `US-*.md`, `ADR-*.md`) and + scans the code tree for `@covers` citations. There is no flat dependencies + projection to preserve. Option (b) would invent a consumer that does not + exist. +- `workflows/actions/reconcile-alignment.md` likewise does not open a + `dependencies.yaml` file. Its references to "dependencies" are conceptual, + not file-bound. +- The only two real `dependencies.yaml` mentions are prose in + `workflows/actions/frame.md` (validation-gate step) and + `workflows/activities/01-frame/README.md` (artifact-directory description). + Both are documentation of *where validation rules live*, not code that opens + a file. Repointing the prose at `meta.yml.validation` and + `meta.yml.relationships` is a textual edit, no runtime impact. +- Option (b) would add a build/check step, a generated file, and a drift + surface (projection out of sync with `meta.yml`) for no consumer benefit. + That is the opposite of the audit's goal. + +If a future consumer ever needs a flat dependencies graph, it can either parse +`meta.yml` directly or compose a projection at read time. That decision belongs +to that consumer, not to the catalog. + +## Alternatives + +| Option | Pros | Cons | Evaluation | +|--------|------|------|------------| +| Generate `dependencies.yaml` per artifact type from `meta.yml` as a build step | Preserves the historical rubric shape | Adds a generated file, a drift surface, and a build/check step with no consumer that needs it | Rejected: no consumer reads the file; cost without benefit | +| Hand-author both `dependencies.yaml` and `meta.yml.relationships` | Matches the historical rubric literally | Two sources of truth; 48 missing files; encodes the same data twice; drift inevitable | Rejected: contradicts single-source goal | +| **Ratify `meta.yml.relationships` as canonical and drop `dependencies.yaml` from the rubric** | One source of truth; matches what the catalog actually ships; no generation step; rubric stops lying about expected files | Existing prose in `frame.md` and `01-frame/README.md` must be updated in the same change | **Selected: smallest sufficient alignment** | + +## Consequences + +| Type | Impact | +|------|--------| +| Positive | Catalog has one place to look for artifact-type dependencies and one place to look for validation rules: `meta.yml`. | +| Positive | The rubric stops asking for a file that does not exist; the 48/48 absence reported by the audit is no longer a violation. | +| Positive | Workflow prose (`workflows/actions/frame.md`, `workflows/activities/01-frame/README.md`) is repointed at the file that actually carries validation rules, removing a documentation-to-reality gap. | +| Positive | No generated projection means no projection-drift class of failure to defend against. | +| Negative | The rubric line in `workflows/artifact-schema.md` ("Recommended fields") and the prose in `workflows/actions/frame.md` / `workflows/activities/01-frame/README.md` must be updated in lockstep with this ADR; any future doc that re-introduces `dependencies.yaml` must be rejected as drift. | +| Neutral | Instance-level dependencies in `ddx.depends_on` frontmatter are unaffected; this ADR is about type-level rubric, not instance-level graph. | + +## Risks + +| Risk | Prob | Impact | Mitigation | +|------|------|--------|------------| +| A future workflow re-introduces `dependencies.yaml` by force of habit | M | L | Audit gate: a grep for `dependencies.yaml` outside ADR-004 itself should return zero hits after this change | +| A future consumer wants a flat dependencies projection and reinvents it ad hoc | L | M | Consumers may compose a projection from `meta.yml` at read time; no canonical file required | +| The audit's "48/48 absent" finding gets re-reported because the rubric is read out of date | L | L | Update the rubric prose in the same commit as this ADR so the next audit sees the canonical shape | + +## Validation + +| Success Metric | Review Trigger | +|----------------|----------------| +| Grep for `dependencies.yaml` outside ADR-004 returns zero hits in `workflows/` and `scripts/` | A workflow doc or script re-introduces `dependencies.yaml` | +| `workflows/actions/frame.md` validation-gate step references `meta.yml.validation` (not `dependencies.yaml`) | A reviewer cannot tell where validation rules live for an artifact type | +| `workflows/activities/01-frame/README.md` artifact-directory description does not list `dependencies.yaml` | A new artifact type ships a `dependencies.yaml` instead of `meta.yml` updates | +| `workflows/artifact-schema.md` "Recommended fields" no longer implies a separate dependencies file | A consumer ships code that opens `dependencies.yaml` | + +## References + +- [Audit plan 2026-05-30](/artifacts/plan-2026-05-30-artifact-types-and-concerns-audit/) +- [PRD](/artifacts/prd/) +- [Artifact schema](https://github.com/DocumentDrivenDX/helix/blob/main/workflows/artifact-schema.md) +- ADR-002: HELIX Tracker Write Safety Model +- ADR-003: Autonomy Spectrum diff --git a/docs/website/content/artifacts/adr/ADR-005-concern-practices-activity-keyed.md b/docs/website/content/artifacts/adr/ADR-005-concern-practices-activity-keyed.md new file mode 100644 index 00000000..66f9f218 --- /dev/null +++ b/docs/website/content/artifacts/adr/ADR-005-concern-practices-activity-keyed.md @@ -0,0 +1,93 @@ +--- +title: "ADR-005: Concern practices.md is HELIX-activity-keyed" +slug: ADR-005-concern-practices-activity-keyed +weight: 200 +activity: "Design" +source: "02-design/adr/ADR-005-concern-practices-activity-keyed.md" +generated: true +collection: adr +--- + +> **Source identity** (from `02-design/adr/ADR-005-concern-practices-activity-keyed.md`): + +```yaml +ddx: + id: ADR-005 + depends_on: + - helix.prd +``` + +# ADR-005: Concern practices.md is HELIX-activity-keyed + +| Date | Status | Deciders | Related | Confidence | +|------|--------|----------|---------|------------| +| 2026-05-30 | Accepted | HELIX maintainers | concerns library, context-digest assembly | High | + +## Context + +| Aspect | Description | +|--------|-------------| +| Problem | Concern `practices.md` files use two incompatible organizing principles. Some are keyed by HELIX activity (Discover / Frame / Design / Test / Build / Deploy / Iterate), so per-activity context-digest assembly is a mechanical lookup. Others are keyed by topic (Implementation / Quality Gates / etc.), forcing the context assembler to guess which heading belongs to which activity. | +| Current State | Ten concerns are topic-keyed: `auth-local-sessions`, `caching-strategy`, `classic-layered`, `cqrs`, `domain-driven-design`, `enterprise-application-patterns`, `hexagonal-architecture`, `mcp-server`, `onion-architecture`, `sample-data`. The remaining concerns already organize practices by HELIX activity. | +| Requirements | Per-activity context-digest assembly must be a deterministic lookup, not a heuristic. The concerns library must use one consistent organizing principle so that any consumer — context assembler, alignment review, methodology docs — extracts the practices for a given activity by reading the corresponding heading. | + +## Decision + +Every concern's `practices.md` organizes its per-activity content under HELIX +activity headings: **Discover**, **Frame**, **Design**, **Test**, **Build**, +**Deploy**, **Iterate**. Topic-keyed or quality-gates-keyed organization is no +longer acceptable for `practices.md`. + +Activity-keying makes per-activity context-digest assembly a mechanical lookup: +the assembler reads the heading whose name matches the current activity and +pulls the bullets underneath. No guessing, no mapping table, no per-concern +schema. + +The ten topic-keyed concerns named above convert to activity-keying in Phase 3 +of the artifact-types-and-concerns audit. New concerns ship activity-keyed from +day one. + +**Key Points**: HELIX activity headings are the only acceptable top-level +structure in `practices.md` | mechanical per-activity lookup | ten existing +topic-keyed concerns convert in Phase 3 + +## Alternatives + +| Option | Pros | Cons | Evaluation | +|--------|------|------|------------| +| Keep both styles (activity-keyed and topic-keyed) | No conversion work | Context-digest assembly must implement per-concern mapping logic and stay in sync as concerns evolve | Rejected: pushes guesswork into every consumer | +| Topic-keyed everywhere (Implementation / Quality Gates / ...) | Matches a few existing concerns | Forces the assembler to guess which topic applies to which activity; topics differ by concern | Rejected: not mechanically extractable | +| Add a frontmatter mapping table from topic → activity in each `practices.md` | Preserves topic prose | Introduces a per-concern schema to maintain; the heading and the mapping can drift | Rejected: heading is already the natural index | +| **Activity-keyed everywhere; convert the ten topic-keyed concerns in Phase 3** | One consistent structure; mechanical lookup; matches the majority of existing concerns | Requires per-concern restructuring for ten concerns | **Selected: smallest sufficient unification** | + +## Consequences + +| Type | Impact | +|------|--------| +| Positive | Per-activity context-digest assembly becomes a mechanical heading lookup across every concern. | +| Positive | Downstream consumers (alignment review, methodology docs, the website mirror) get a uniform structure to render and reason about. | +| Positive | New concerns have one obvious shape to follow; no per-concern bikeshed about structure. | +| Negative | The ten topic-keyed concerns require restructuring; some topic prose (e.g. cross-cutting Quality Gates) has to be reseated under the activity it applies to. | +| Neutral | The bullet content of each concern is preserved; only the organizing headings change. | + +## Risks + +| Risk | Prob | Impact | Mitigation | +|------|------|--------|------------| +| Conversion silently drops practice content during restructuring | M | M | Phase 3 conversion reviews the diff per concern; bullets must land under exactly one activity heading | +| Cross-cutting content (e.g. Quality Gates) has no natural single activity home | M | M | Allow the same bullet to appear under more than one activity heading when it genuinely applies across activities; prefer the closest activity otherwise | +| New concerns drift back to topic-keying | L | M | Lint or review check rejects `practices.md` whose top-level headings are not from the HELIX activity vocabulary | + +## Validation + +| Success Metric | Review Trigger | +|----------------|----------------| +| Every `practices.md` uses HELIX activity headings as its top-level structure | A `practices.md` lands or merges with topic-keyed headings | +| The ten named concerns are converted in Phase 3 with no lost content | Phase 3 closes with any of the ten still topic-keyed | +| Context-digest assembly extracts per-activity practices by heading lookup, with no per-concern mapping | The assembler grows per-concern conditional logic | + +## References + +- [PRD](/artifacts/prd/) +- Phase 1A plan: `docs/helix/02-design/plan-2026-05-30-artifact-types-and-concerns-audit.md` +- Concerns library: `workflows/concerns/` diff --git a/docs/website/content/artifacts/adr/ADR-006-concern-boundary-lives-once.md b/docs/website/content/artifacts/adr/ADR-006-concern-boundary-lives-once.md new file mode 100644 index 00000000..02aabfbc --- /dev/null +++ b/docs/website/content/artifacts/adr/ADR-006-concern-boundary-lives-once.md @@ -0,0 +1,124 @@ +--- +title: "ADR-006: Concern boundary lives once, in concern.md" +slug: ADR-006-concern-boundary-lives-once +weight: 210 +activity: "Design" +source: "02-design/adr/ADR-006-concern-boundary-lives-once.md" +generated: true +collection: adr +--- + +> **Source identity** (from `02-design/adr/ADR-006-concern-boundary-lives-once.md`): + +```yaml +ddx: + id: ADR-006 + depends_on: + - helix.prd +``` + +# ADR-006: Concern boundary lives once, in concern.md + +| Date | Status | Deciders | Related | Confidence | +|------|--------|----------|---------|------------| +| 2026-05-30 | Accepted | HELIX maintainers | plan-2026-05-30-artifact-types-and-concerns-audit, ADR-004 (dependencies encoding), ADR-005 (practices format) | High | + +## Context + +| Aspect | Description | +|--------|-------------| +| Problem | The current concern file shape mandates three sections that naturally restate the same boundary: `concern.md`'s `Boundary` section, `practices.md`'s `Boundary with neighbors` section, and the `Constraints` / `Drift Signals` / `Quality Gates` sections that re-encode the same MUSTs in three different formats. Restating the boundary in three places is the structural cause of the verbosity flagged across the catalog — not editorial style. | +| Current State | The 2026-05-30 audit rated 11 of 48 artifact types and 22 of 49 concerns `verbose`. The concerns trace uniformly to this restatement pattern: the audit found `authorization-model`, `multi-tenancy`, `resilience`, `enterprise-integration-patterns`, `relational-data-modeling`, `verification`, `usage-metering`, `ux-radix`, `twelve-factor`, `event-sourcing`, `deployment-topology` all carrying the same Boundary text in three places. Editorial trims partially regress without a shape fix because the contract still asks for three boundary sections. The audit also surfaced two adjacent restatement vectors — `meta.yml` carrying taxonomies and process metadata that duplicate prompt/template content, and prompts/templates each carrying their own checklist of the same rules — that an editorial-only pass would not address. | +| Requirements | The catalog contract must define section ownership so the boundary is stated once, the other sections reference-not-restate it, and a schema validator can flag restatement as drift instead of relying on editorial review. The contract must also fold in the editorial rules from the dropped concision phase so the same contract that eliminates structural restatement also eliminates the adjacent vectors. | + +## Decision + +Every concern's boundary lives canonically in `concern.md`'s `Boundary` +section, and **only** there. The audit-named `Boundary with neighbors` +section in `practices.md` is removed; the `Constraints`, `Drift Signals`, +and `Quality Gates` sections that remain (in either file, per the +practices-format decision in ADR-005) **reference** the canonical boundary +rather than restating it. + +Restatement of boundary prose outside `concern.md`'s `Boundary` section is +now a **drift signal** the catalog schema validator (introduced in Phase 1 +Track B) can detect and flag. + +We also fold the editorial rules from the dropped concision phase into this +ADR so the contract that fixes the structural restatement also fixes the +adjacent vectors: + +1. **`meta.yml` is for machine validation + relationships only.** It does + not carry taxonomies, process metadata, decision frameworks, success + indicators, engagement levels, or other content that duplicates prompt + or template prose. Anything in `meta.yml` that is not consumed by a + validator or relationship resolver is drift. +2. **Prompt Quality Checklist and template Review Checklist are not both + present.** The catalog picks one home per artifact type; the other is + removed. (Default home: template Review Checklist for human-authored + artifacts; prompt Quality Checklist for generator-authored artifacts. + Per-artifact choice is recorded in `meta.yml`.) +3. **Tutorial-grade code blocks move to resource docs.** Full Dockerfiles, + complete Hugo configs, large CSS blocks, multi-step tmux scripts, and + similar long-form examples do not live in `concern.md` or + `practices.md`. They live under `docs/resources/` and the concern + references them by path. + +Each of these is enforceable by the same validator that flags boundary +restatement: presence of disallowed `meta.yml` keys, presence of both +checklists, or code blocks exceeding a size threshold become drift +signals. + +**Key Points**: boundary stated once in `concern.md` | other sections +reference-not-restate | restatement is a validator-flagged drift signal | +`meta.yml` is machine-only | one checklist per artifact | tutorial code +lives in resource docs + +## Alternatives + +| Option | Pros | Cons | Evaluation | +|--------|------|------|------------| +| Keep three boundary sections; rely on editorial review to keep them in sync | No contract change; no validator work | The 2026-05-30 audit shows editorial review does not keep them in sync — 22 concerns drifted into restatement; partial trims regress because the contract still asks for three sections | Rejected: this is the status quo the audit flagged | +| Put the boundary in `practices.md`'s `Boundary with neighbors` instead and have `concern.md`'s `Boundary` reference it | Keeps the neighbor-comparison framing | `concern.md` is the entry document a reader opens first; making it reference forward is worse ergonomics; the `practices.md` boundary section is the one that drifted, not `concern.md`'s | Rejected: wrong canonical home | +| Remove the boundary concept entirely; rely on Constraints / Drift Signals / Quality Gates | One fewer section type | The boundary is the load-bearing scope statement; the MUSTs are derived from it. Removing the boundary makes the MUSTs unmoored | Rejected: loses the source of truth | +| **Boundary lives once in `concern.md`; everything else references; validator flags restatement; fold editorial rules into the same contract** | Eliminates structural cause of restatement; mechanizable; one contract covers all three vectors | Requires validator work and a mechanical edit phase to remove the restated prose | **Selected: smallest sufficient structural fix** | + +## Consequences + +| Type | Impact | +|------|--------| +| Positive | The Phase 3 mechanical edits become a deterministic translation — remove the `Boundary with neighbors` section and rewrite Constraints / Drift Signals / Quality Gates as references — rather than a judgment call. | +| Positive | Future concerns can pass the catalog validator only if the boundary is stated once; the verbose-by-restatement pattern cannot reintroduce silently. | +| Positive | The validator that enforces single-boundary also enforces the `meta.yml` / dual-checklist / tutorial-code rules, so the dropped concision phase's intent is preserved without a separate editorial pass. | +| Positive | The catalog contract becomes one coherent policy — file shape, machine metadata, prose ownership, and resource separation are all enforced by the same validator. | +| Negative | Every existing concern needs the Phase 3 mechanical edit (remove `Boundary with neighbors`, rewrite three sections to reference the canonical boundary). | +| Negative | Several `meta.yml` files lose substantial content (the taxonomies and process metadata that duplicate prompt/template); this is a one-time migration. | +| Negative | The validator must define what "restatement" means precisely enough to catch real drift without false positives on legitimate cross-reference prose. | +| Neutral | The `practices.md` file remains; its body (Design / Implementation / MUST / SHOULD / Quality Gates, or the activity-keyed form per ADR-005) keeps its non-boundary content. | + +## Risks + +| Risk | Prob | Impact | Mitigation | +|------|------|--------|------------| +| Validator definition of "restatement" produces false positives on legitimate cross-references | M | M | Start with structural checks (presence of a `Boundary with neighbors` H2; `meta.yml` keys outside the allowlist; both checklists present) before attempting prose-similarity detection | +| Phase 3 edits remove prose readers actually relied on for the neighbor comparison | L | M | The canonical `Boundary` section can grow to subsume any load-bearing neighbor language; the rule is "stated once," not "stated minimally" | +| The `meta.yml` allowlist is too narrow and breaks existing tooling | M | M | Inventory consumers of `meta.yml` keys before defining the allowlist; the allowlist is the union of consumed keys | +| Tutorial code moved to `docs/resources/` loses context that made it useful in-line | L | L | Concerns reference the resource doc by path with a one-line summary of what the resource demonstrates | +| Choosing prompt vs template checklist per artifact reopens the orthogonality discussion the catalog is trying to close | L | M | Record the default rule (template checklist for human-authored, prompt for generator-authored) and require an explicit override in `meta.yml` only when deviating | + +## Validation + +| Success Metric | Review Trigger | +|----------------|----------------| +| The catalog schema validator flags any concern with a `Boundary with neighbors` H2 outside `concern.md`'s `Boundary` section | A new concern lands with restated boundary prose without the validator failing | +| Every `meta.yml` in the catalog contains only keys on the consumer-derived allowlist | A `meta.yml` key is added that no validator or relationship resolver reads | +| No artifact type ships both a prompt Quality Checklist and a template Review Checklist | A PR introduces or restores the duplicate checklist | +| Concern files contain no tutorial-grade code blocks (size threshold honored); tutorial content lives under `docs/resources/` and is referenced by path | A concern grows a multi-screen code example inline | +| Phase 3's mechanical edit pass removes the `Boundary with neighbors` section from every concern and rewrites Constraints / Drift Signals / Quality Gates as references, without regressing the canonical `Boundary` content | The audit re-run still flags the restatement pattern after Phase 3 lands | + +## References + +- [Plan: artifact-types-and-concerns audit (2026-05-30)](/artifacts/plan-2026-05-30-artifact-types-and-concerns-audit/) +- ADR-004: Dependencies encoding (Phase 1 catalog contract) +- ADR-005: Practices format (Phase 1 catalog contract) +- [PRD](/artifacts/prd/) diff --git a/docs/website/content/artifacts/adr/ADR-007-empty-adr-references-headers-dropped.md b/docs/website/content/artifacts/adr/ADR-007-empty-adr-references-headers-dropped.md new file mode 100644 index 00000000..e6bd748f --- /dev/null +++ b/docs/website/content/artifacts/adr/ADR-007-empty-adr-references-headers-dropped.md @@ -0,0 +1,83 @@ +--- +title: "ADR-007: Empty ADR References Headers Are Dropped; Required Only When a Concern Cites a Specific ADR" +slug: ADR-007-empty-adr-references-headers-dropped +weight: 220 +activity: "Design" +source: "02-design/adr/ADR-007-empty-adr-references-headers-dropped.md" +generated: true +collection: adr +--- + +> **Source identity** (from `02-design/adr/ADR-007-empty-adr-references-headers-dropped.md`): + +```yaml +ddx: + id: ADR-007 + depends_on: + - helix.prd +``` + +# ADR-007: Empty ADR References Headers Are Dropped; Required Only When a Concern Cites a Specific ADR + +| Date | Status | Deciders | Related | Confidence | +|------|--------|----------|---------|------------| +| 2026-05-30 | Accepted | HELIX maintainers | concerns library, schema validator | High | + +## Context + +| Aspect | Description | +|--------|-------------| +| Problem | Many concerns ship an `ADR References` section header with no entries beneath it. The empty header creates a false promise of governing decisions that do not exist, and pads the artifact with structure that carries no information. | +| Current State | 14 concerns currently ship an empty `ADR References` header: `a11y-wcag-aa`, `demo-asciinema`, `demo-playwright`, `design-patterns-gof`, `e2e-kind`, `e2e-playwright`, `go-std`, `hugo-hextra`, `i18n-icu`, `k8s-kind`, `o11y-otel`, `python-uv`, `rust-cargo`, `scala-sbt`. The schema validator does not require the section to be present. | +| Requirements | The concerns library must not advertise references that do not exist. The section should appear only when it carries real ADR citations, and should be omitted when it would otherwise be empty. The schema validator must continue to treat the section as optional. | + +## Decision + +An `ADR References` section header in a concern is included **only when the +concern actually cites one or more ADRs**. Empty headers are removed by Phase 3 +mechanical edits across the 14 concerns enumerated above. The schema validator +does not require this section; presence is driven by content, not by template +shape. + +Concerns that genuinely govern decisions via ADRs keep the section and list the +ADRs they cite. Concerns with no real ADR citations omit the header entirely. + +**Key Points**: header presence follows content | empty headers removed in +Phase 3 | schema validator keeps the section optional + +## Alternatives + +| Option | Pros | Cons | Evaluation | +|--------|------|------|------------| +| Keep empty `ADR References` headers as a template slot | Uniform structure across concerns | False promise of citations; padding without information; invites cargo-cult copy of the empty section into new concerns | Rejected: structure without content is noise | +| Require every concern to cite at least one ADR | Forces explicit governance | Most concerns are not governed by an ADR; would manufacture spurious references | Rejected: not all concerns need ADR backing | +| **Drop empty headers; require the section only when ADRs are cited** | Removes false promise; keeps real citations visible; matches existing validator behavior | Requires a one-time mechanical edit pass across 14 concerns | **Selected: smallest change that aligns shape with content** | + +## Consequences + +| Type | Impact | +|------|--------| +| Positive | Concerns no longer advertise references that do not exist. | +| Positive | Real ADR citations remain visible and unambiguous in the concerns that have them. | +| Positive | New concerns inherit a "populate or omit" rule that prevents the empty header from reappearing. | +| Negative | Phase 3 must touch 14 concern files to remove the empty headers. | +| Neutral | The schema validator is unchanged; the section stays optional. | + +## Risks + +| Risk | Prob | Impact | Mitigation | +|------|------|--------|------------| +| Future concerns reintroduce the empty header by copy-paste | M | L | Phase 3 edits remove existing instances; reviewers reject empty headers on new concerns | +| A real ADR citation is accidentally removed alongside the empty headers | L | M | Phase 3 mechanical edits target only the enumerated 14 concerns where the section is empty | + +## Validation + +| Success Metric | Review Trigger | +|----------------|----------------| +| None of the 14 enumerated concerns ships an empty `ADR References` header after Phase 3 | An empty `ADR References` header appears in any concern | +| Concerns that cite ADRs retain the section with the citations intact | A real ADR citation is dropped by the Phase 3 edits | + +## References + +- [PRD](/artifacts/prd/) +- [Plan: 2026-05-30 Artifact Types and Concerns Audit](/artifacts/plan-2026-05-30-artifact-types-and-concerns-audit/) diff --git a/docs/website/content/artifacts/adr/ADR-008-data-prd-is-prd-kind-variant.md b/docs/website/content/artifacts/adr/ADR-008-data-prd-is-prd-kind-variant.md new file mode 100644 index 00000000..2c651a7a --- /dev/null +++ b/docs/website/content/artifacts/adr/ADR-008-data-prd-is-prd-kind-variant.md @@ -0,0 +1,89 @@ +--- +title: "ADR-008: data-PRD Is a Kind-Switch Variant of PRD, Not a Sibling Artifact" +slug: ADR-008-data-prd-is-prd-kind-variant +weight: 230 +activity: "Design" +source: "02-design/adr/ADR-008-data-prd-is-prd-kind-variant.md" +generated: true +collection: adr +--- + +> **Source identity** (from `02-design/adr/ADR-008-data-prd-is-prd-kind-variant.md`): + +```yaml +ddx: + id: ADR-008 + depends_on: + - helix.prd +``` + +# ADR-008: data-PRD Is a Kind-Switch Variant of PRD, Not a Sibling Artifact + +| Date | Status | Deciders | Related | Confidence | +|------|--------|----------|---------|------------| +| 2026-05-30 | Accepted | HELIX maintainers | artifact-types catalog, prd, data-prd | High | + +## Context + +| Aspect | Description | +|--------|-------------| +| Problem | The `data-prd` artifact type is a near-duplicate of `prd` with data-product framing. Its `required_sections` drift from the `prd` template, the boundary between the two artifact types is not load-bearing, and the catalog carries an orthogonality defect: two artifact-type directories that differ only in framing, not in role. | +| Current State | `workflows/activities/01-frame/artifacts/prd/` and `workflows/activities/01-frame/artifacts/data-prd/` ship as sibling artifact-type directories. Both serve the framing role of "product requirements document"; `data-prd` only adjusts vocabulary toward data products. The two trees have diverged in required sections without a substantive reason. | +| Requirements | The catalog must represent one PRD role with explicit variant framing rather than two parallel artifact types. The variant must remain discoverable for data-product framing without forcing every PRD to carry data-product vocabulary. Downstream consumers that reference `data-prd` must continue to resolve. | + +## Decision + +`data-prd` collapses into `prd` as a `kind: data` variant. The PRD artifact +gains a `kind` field (default `product`); when set to `data`, the prompt and +template guidance shift to data-product framing under conditional headings. +The current `data-prd` artifact-type directory is deprecated and will be +removed in Phase 3. + +The PRD artifact owns one role — framing product (or data-product) +requirements — and exposes the variant through a kind switch on its meta. The +shape of the artifact is unified; the framing is parameterized. + +**Key Points**: one PRD artifact with a `kind` switch | `kind: product` +default, `kind: data` for data products | `data-prd` directory removed in +Phase 3 with alias for downstream consumers + +## Alternatives + +| Option | Pros | Cons | Evaluation | +|--------|------|------|------------| +| Keep `data-prd` as a sibling artifact type | No migration work; existing references unchanged | Perpetuates orthogonality defect; required-section drift continues; catalog advertises a boundary that is not load-bearing | Rejected: the boundary does not carry weight | +| Merge `data-prd` into `prd` with no variant marker | Simplest collapse | Loses data-product framing entirely; existing data-PRD consumers lose guidance | Rejected: framing variant has real value | +| **Collapse `data-prd` into `prd` as a `kind: data` variant** | Unifies the role; preserves the framing as a parameterized variant; eliminates the duplicate directory; lets the template carry conditional guidance | Phase 3 must edit `prd` meta/prompt/template and remove the `data-prd` directory; downstream references to `data-prd` must be redirected or aliased | **Selected: smallest change that resolves the defect while preserving the framing variant** | + +## Consequences + +| Type | Impact | +|------|--------| +| Positive | The artifact-type catalog represents one PRD role instead of two near-duplicates. | +| Positive | Required-section drift between `prd` and `data-prd` ends; one template governs both framings. | +| Positive | Data-product framing remains discoverable through an explicit `kind: data` switch rather than a parallel directory. | +| Positive | New PRD-shaped variants (if any) can be added as additional `kind` values without forking the directory. | +| Negative | Phase 3 must extend `prd` meta/prompt/template, delete the `data-prd` directory, and update or alias every reference under `docs/helix/` and `workflows/`. | +| Neutral | Existing PRDs default to `kind: product` and require no content change. | + +## Risks + +| Risk | Prob | Impact | Mitigation | +|------|------|--------|------------| +| Downstream consumers still resolve `data-prd` by path | M | M | Add an alias in any downstream consumer that resolves artifact-type paths; update references in the same Phase 3 pass | +| Conditional kind-specific guidance in the template becomes hard to read | M | M | Keep kind-specific guidance under clearly labeled conditional headings; avoid interleaving product and data framing line-by-line | +| New PRD variants accumulate via the kind switch without governance | L | M | Document the kind enum in `prd` meta with the supported values; new values require an ADR | + +## Validation + +| Success Metric | Review Trigger | +|----------------|----------------| +| `workflows/activities/01-frame/artifacts/data-prd/` is removed after Phase 3 | A `data-prd` artifact-type directory remains after Phase 3 | +| `prd` meta declares a `kind` enum with `product` (default) and `data` values | The `prd` artifact ships without a `kind` field after Phase 3 | +| `prd` prompt and template carry kind-specific guidance under conditional headings | Data-product framing is lost from the unified PRD artifact | +| All references to `data-prd` under `docs/helix/` and `workflows/` resolve to `prd` with the kind switch | A reference to `data-prd` resolves to a missing directory after Phase 3 | + +## References + +- [PRD](/artifacts/prd/) +- [Plan: 2026-05-30 Artifact Types and Concerns Audit](/artifacts/plan-2026-05-30-artifact-types-and-concerns-audit/) diff --git a/docs/website/content/artifacts/adr/ADR-009-acceptance-criteria-ownership.md b/docs/website/content/artifacts/adr/ADR-009-acceptance-criteria-ownership.md new file mode 100644 index 00000000..4814c51d --- /dev/null +++ b/docs/website/content/artifacts/adr/ADR-009-acceptance-criteria-ownership.md @@ -0,0 +1,88 @@ +--- +title: "ADR-009: Acceptance Criteria Live in User-Stories; Feature-Specification Owns Functional Areas and Decomposition" +slug: ADR-009-acceptance-criteria-ownership +weight: 240 +activity: "Design" +source: "02-design/adr/ADR-009-acceptance-criteria-ownership.md" +generated: true +collection: adr +--- + +> **Source identity** (from `02-design/adr/ADR-009-acceptance-criteria-ownership.md`): + +```yaml +ddx: + id: ADR-009 + depends_on: + - helix.prd +``` + +# ADR-009: Acceptance Criteria Live in User-Stories; Feature-Specification Owns Functional Areas and Decomposition + +| Date | Status | Deciders | Related | Confidence | +|------|--------|----------|---------|------------| +| 2026-05-30 | Accepted | HELIX maintainers | user-stories, feature-specification | High | + +## Context + +| Aspect | Description | +|--------|-------------| +| Problem | The rule "AC-IDs live in user-stories" is restated five times across the `user-stories` artifact (meta `quality_checks`, meta `automated_checks`, prompt section, prompt blocking checklist, template) and partly mirrored on `feature-specification`. The boundary "FEAT owns FR-n; stories own AC-IDs" is asserted in multiple places but is not load-bearing in any single one, which makes it easy for a future edit to drop the rule on one side without noticing. | +| Current State | `feature-specification` includes acceptance-criteria guidance in its prompt and template (covering happy paths, errors, edge cases with observable outcomes) alongside its FR-n and functional-area material. `user-stories` carries the canonical AC-ID rule but duplicates it across five surfaces. The split of ownership between the two artifacts is implicit rather than declared. | +| Requirements | The HELIX frame phase must have exactly one artifact that owns acceptance-criteria definition and AC-IDs. The boundary must be stated once, load-bearing, and survive future edits. Feature-specification must remain the home of functional areas and the decomposition test (which stories the feature decomposes into) without restating AC material. | + +## Decision + +Acceptance criteria and AC-IDs live exclusively in `user-stories`. +`feature-specification` declares FR-IDs, functional areas, and the +decomposition test (which user stories the feature decomposes into) but does +**not** define or restate acceptance criteria. + +Phase 3 will remove AC-ID and acceptance-criteria restatement from +`feature-specification`'s prompt and template. `user-stories` keeps the single +canonical AC-ID rule, with the meta-quality-check duplication trimmed so the +rule lives in one place rather than five. + +**Key Points**: AC-IDs owned by `user-stories` | `feature-specification` owns +FR-IDs, functional areas, and decomposition | rule stated once, not echoed +across surfaces + +## Alternatives + +| Option | Pros | Cons | Evaluation | +|--------|------|------|------------| +| Keep AC material on both artifacts and rely on cross-references | Uniform appearance; each artifact looks self-contained | Five-way restatement on `user-stories` plus partial mirror on `feature-specification`; no single load-bearing statement; future edits silently drift one side | Rejected: duplication without authority is the current failure mode | +| Move AC ownership to `feature-specification` instead | Feature is the higher-level container; AC near FRs reads naturally | Stories become weaker as test contracts; AC-IDs lose their natural scope (US-``-AC``); breaks the decomposition direction (feature -> stories -> AC) | Rejected: AC-ID identity is story-scoped, not feature-scoped | +| **AC-IDs on `user-stories` only; `feature-specification` owns FR-IDs and decomposition** | Single owner per concept; load-bearing rule; preserves the FR -> story -> AC chain; matches how AC-IDs are already scoped | Requires Phase 3 mechanical edits to `feature-specification` prompt/template and trimming `user-stories` meta duplication | **Selected: smallest change that makes the boundary load-bearing** | + +## Consequences + +| Type | Impact | +|------|--------| +| Positive | The "AC-IDs live in user-stories" rule is stated once and is load-bearing rather than echoed five ways. | +| Positive | `feature-specification` focuses on FR-IDs, functional areas, and the decomposition test, which is its actual job. | +| Positive | The FR -> story -> AC decomposition direction is visible in the artifact shapes themselves. | +| Positive | Future edits cannot silently drift the boundary; there is one place to change. | +| Negative | Phase 3 must touch `feature-specification`'s prompt and template to remove acceptance-criteria material, and trim `user-stories` meta to collapse the five-way restatement. | +| Neutral | The FR-n / AC-ID identifier schemes themselves are unchanged. | + +## Risks + +| Risk | Prob | Impact | Mitigation | +|------|------|--------|------------| +| `feature-specification` authors continue to write AC inline by habit | M | M | Phase 3 prompt edits redirect AC material to `user-stories`; reviewers reject AC in feature-specification | +| Removing AC guidance from `feature-specification` weakens feature-level testability framing | L | M | Decomposition test stays on feature-specification and points reviewers at the story-level AC for testability | +| `user-stories` meta trimming drops a check that was actually catching defects | L | M | Phase 3 keeps the single canonical AC-ID rule; only redundant restatements are removed | + +## Validation + +| Success Metric | Review Trigger | +|----------------|----------------| +| `feature-specification` prompt and template contain no AC-ID or acceptance-criteria definitions after Phase 3 | Acceptance-criteria material reappears in `feature-specification` | +| `user-stories` states the AC-ID rule in exactly one canonical place | The AC-ID rule is restated across multiple surfaces of `user-stories` again | +| `feature-specification` carries the decomposition test (which stories the feature decomposes into) and FR-IDs as its primary content | A feature is filed without the decomposition test or FR-IDs | + +## References + +- [PRD](/artifacts/prd/) +- [Plan: 2026-05-30 Artifact Types and Concerns Audit](/artifacts/plan-2026-05-30-artifact-types-and-concerns-audit/) diff --git a/docs/website/content/artifacts/adr/ADR-010-feature-registry-parking-lot-handoff.md b/docs/website/content/artifacts/adr/ADR-010-feature-registry-parking-lot-handoff.md new file mode 100644 index 00000000..2acc8848 --- /dev/null +++ b/docs/website/content/artifacts/adr/ADR-010-feature-registry-parking-lot-handoff.md @@ -0,0 +1,82 @@ +--- +title: "ADR-010: feature-registry and parking-lot stay separate with an explicit handoff" +slug: ADR-010-feature-registry-parking-lot-handoff +weight: 250 +activity: "Design" +source: "02-design/adr/ADR-010-feature-registry-parking-lot-handoff.md" +generated: true +collection: adr +--- +# ADR-010: feature-registry and parking-lot stay separate with an explicit handoff + +| Date | Status | Deciders | Related | Confidence | +|------|--------|----------|---------|------------| +| 2026-05-30 | Accepted | HELIX maintainers | feature-registry, parking-lot, frame-phase catalog | High | + +## Context + +| Aspect | Description | +|--------|-------------| +| Problem | The artifact-type audit flagged `feature-registry` and `parking-lot` as similarly-shaped registries in the frame phase, raising the question of whether they should collapse into a single registry distinguished by a status enum. | +| Current State | `feature-registry` is the source of truth for accepted features (`FEAT-XXX`, owners, priorities, status progression from `draft` through `deployed`/`deprecated`/`cancelled`). `parking-lot` is a separate registry for deferred or future work, keyed by `Source` and `Revisit Trigger` and explicitly outside current scope. They share a `type: registry` classification and live in the same activity, but encode different lifecycle states and serve different operator decisions. | +| Requirements | The catalog must either (a) merge the two into one registry with a status discriminator, or (b) keep them separate with a documented handoff so the boundary the operator triages on stays visible and tracker-relevant. | + +## Decision + +`feature-registry` and `parking-lot` remain separate artifacts. + +`parking-lot` is the staging area for deferred or future work — items kept +visible but outside current scope until a recorded `Revisit Trigger` fires. +`feature-registry` is the source of truth for accepted features that are in +the active pipeline (`draft` → `specified` → `designed` → `in_test` → +`in_build` → `built` → `deployed`, or terminal `deprecated`/`cancelled`). + +Handoff between the two is explicit: a parking-lot entry promotes to +feature-registry when its revisit criteria are met. The promotion is a +recorded transition (assign next sequential `FEAT-XXX`, set initial status, +link back to the parking-lot source), not an implicit merge or status flip +inside a single registry. + +**Key Points**: deferred vs accepted is a real lifecycle boundary | promotion +is a recorded transition, not an implicit merge | the operator triages on the +boundary, so the boundary must be visible in the artifact shape + +## Alternatives + +| Option | Pros | Cons | Evaluation | +|--------|------|------|------------| +| Collapse into one registry with a `deferred` status value | One file, one schema, one prompt; removes "similar shape" duplication | Loses the triage boundary the operator uses; mixes accepted features (with owners, priorities, dependencies) with deferred items (with revisit triggers); forces every consumer to filter by status before reading | Rejected: collapses a real lifecycle distinction | +| Keep separate, leave handoff implicit (current state) | No new procedure to document | Promotion happens ad-hoc; no recorded link between parked source and registered feature; the audit's "same shape" concern persists | Rejected: the handoff is the thing that makes separation safe | +| **Keep separate, document an explicit promotion procedure** | Preserves the lifecycle boundary; makes the transition auditable; addresses the audit's concern without merging | Requires Phase 3 to document the promotion procedure once and cross-reference it from parking-lot | **Selected: smallest sufficient fix that keeps the boundary real** | + +## Consequences + +| Type | Impact | +|------|--------| +| Positive | The deferred/accepted boundary stays visible in the artifact shape, matching how operators triage. | +| Positive | Promotion becomes a recorded transition with a back-link, so traceability survives the handoff. | +| Positive | The audit's "same shape" concern is resolved by procedure, not by losing structure. | +| Negative | Phase 3 owes the promotion procedure write-up in `feature-registry`, with a cross-reference from `parking-lot`. | +| Neutral | Both artifacts keep their existing `type: registry` classification and their existing prompts/templates. | + +## Risks + +| Risk | Prob | Impact | Mitigation | +|------|------|--------|------------| +| Promotion procedure documented in one place but not cross-referenced from the other | M | M | Phase 3 acceptance requires both ends — feature-registry owns the procedure, parking-lot links to it | +| Operators bypass the procedure and recreate parked items in the registry without back-links | M | M | The promotion step records the parking-lot source as the first dependency/trace link on the new FEAT-XXX | +| Future audit re-raises the "same shape" question after the procedure rots | L | L | This ADR is the durable answer; future audits cite it instead of relitigating | + +## Validation + +| Success Metric | Review Trigger | +|----------------|----------------| +| feature-registry documents the promotion procedure exactly once | Procedure is duplicated in parking-lot rather than cross-referenced | +| parking-lot's prompt cross-references the promotion procedure | A parked item is promoted without a recorded back-link to its parking-lot source | +| Promoted features carry a recorded link back to the parking-lot entry that sourced them | Operator triage cannot distinguish "deferred" from "accepted" at a glance | + +## References + +- [Plan: artifact-types and concerns audit](/artifacts/plan-2026-05-30-artifact-types-and-concerns-audit/) +- `workflows/activities/01-frame/artifacts/feature-registry/` +- `workflows/activities/01-frame/artifacts/parking-lot/` diff --git a/docs/website/content/artifacts/adr/ADR-011-ux-radix-owns-shadcn-prescription.md b/docs/website/content/artifacts/adr/ADR-011-ux-radix-owns-shadcn-prescription.md new file mode 100644 index 00000000..067bf3fe --- /dev/null +++ b/docs/website/content/artifacts/adr/ADR-011-ux-radix-owns-shadcn-prescription.md @@ -0,0 +1,86 @@ +--- +title: "ADR-011: ux-radix owns Radix and shadcn prescriptions; react-nextjs references" +slug: ADR-011-ux-radix-owns-shadcn-prescription +weight: 260 +activity: "Design" +source: "02-design/adr/ADR-011-ux-radix-owns-shadcn-prescription.md" +generated: true +collection: adr +--- +# ADR-011: ux-radix owns Radix and shadcn prescriptions; react-nextjs references + +| Date | Status | Deciders | Related | Confidence | +|------|--------|----------|---------|------------| +| 2026-05-30 | Accepted | HELIX maintainers | ux-radix, react-nextjs, concerns | High | + +## Context + +| Aspect | Description | +|--------|-------------| +| Problem | Both the `react-nextjs` concern and the `ux-radix` concern prescribe Radix primitives and the shadcn component library. This double-prescription confuses the concern boundary: which concern owns component-library choice, and which one is being referenced? Audits surface the duplication as drift rather than as a deliberate cross-reference. | +| Current State | `react-nextjs` (framework slot) and `ux-radix` (interaction-primitives slot) each independently prescribe shadcn/Radix in their `concern.md` and `practices.md`. There is no single canonical owner, and downstream consumers cannot tell which concern's prescription is authoritative. | +| Requirements | Component-library prescription must live in exactly one concern, with other concerns referencing the owner rather than re-prescribing. The owner should be the concern whose subject matter is interaction primitives and component libraries. | + +## Decision + +The `ux-radix` concern owns the Radix primitives and shadcn component-library +prescription. `react-nextjs` references `ux-radix` for UI primitives and does +NOT prescribe shadcn directly. + +`ux-radix` is the interaction-primitives concern; UI and component-library +prescription naturally lives there. `react-nextjs` is the framework slot +occupant — it should describe the framework (Next.js conventions, routing, +rendering model, build/runtime), not the component library it happens to pair +with. Removing component-library prescription from `react-nextjs` keeps the +framework concern focused on framework-shaped decisions. + +This follows the "concern boundary lives once" rule established in ADR-006: +every prescription has exactly one canonical owner, and other concerns +reference rather than re-state. + +**Key Points**: ux-radix is the canonical owner of Radix + shadcn prescription +| react-nextjs references ux-radix instead of re-prescribing | framework +concerns describe frameworks, not component libraries + +## Alternatives + +| Option | Pros | Cons | Evaluation | +|--------|------|------|------------| +| Keep double-prescription in both concerns | No edits required | Violates ADR-006 single-owner rule; auditors cannot tell which prescription is canonical; drift between the two copies is inevitable | Rejected: known source of confusion | +| Move ownership to `react-nextjs` and have `ux-radix` reference it | Co-locates component-library choice with the framework that pairs with it | Conflates framework slot with interaction-primitives slot; forces every framework concern to re-prescribe its own component library; ux-radix loses its subject matter | Rejected: wrong slot for the prescription | +| **Move ownership to `ux-radix`; `react-nextjs` references it** | Matches concern subject matter (interaction primitives own component libraries); keeps framework concerns focused; preserves slot boundaries | Requires Phase 3 edits to `react-nextjs` to remove or convert its shadcn prescription | **Selected: aligns ownership with concern subject** | + +## Consequences + +| Type | Impact | +|------|--------| +| Positive | Single canonical owner for shadcn/Radix prescription; no duplication to drift between. | +| Positive | `react-nextjs` becomes a focused framework concern instead of a mixed framework + component-library concern. | +| Positive | Future framework concerns (e.g. other React meta-frameworks) can reference `ux-radix` without re-prescribing the component library. | +| Positive | Audit drift signal becomes meaningful: any future shadcn/Radix mention outside `ux-radix` is a real cross-reference, not an accidental duplicate. | +| Negative | Phase 3 must edit `react-nextjs` to remove the shadcn/Radix prescription and replace it with a cross-reference. | +| Negative | Downstream artifacts that cite `react-nextjs` for component-library choice must be updated to cite `ux-radix` instead. | +| Neutral | The pairing of Next.js with shadcn/Radix in practice is unchanged — only the ownership of the prescription moves. | + +## Risks + +| Risk | Prob | Impact | Mitigation | +|------|------|--------|------------| +| Phase 3 edits to `react-nextjs` leave dangling shadcn references | M | M | Audit `react-nextjs/concern.md` and `practices.md` for any remaining shadcn/Radix prescriptions after the edit | +| Other framework concerns silently re-prescribe shadcn | L | M | Drift checker scans non-`ux-radix` concerns for shadcn/Radix prescription language | +| Cross-reference from `react-nextjs` to `ux-radix` is too vague to be useful | M | L | Make the reference explicit: name `ux-radix` and the specific primitives concern, not a generic "see UI concerns" pointer | + +## Validation + +| Success Metric | Review Trigger | +|----------------|----------------| +| Only `ux-radix` prescribes shadcn/Radix; all other concerns reference it | A non-`ux-radix` concern re-prescribes shadcn or Radix primitives | +| `react-nextjs` describes Next.js framework concerns without component-library prescription | `react-nextjs` adds component-library or design-system prescription | +| Downstream artifacts that need component-library choice cite `ux-radix` | An artifact cites `react-nextjs` for shadcn/Radix choice | + +## References + +- ADR-006: Concern boundary lives once +- workflows/concerns/ux-radix/concern.md +- workflows/concerns/react-nextjs/concern.md +- docs/helix/02-design/plan-2026-05-30-artifact-types-and-concerns-audit.md diff --git a/docs/website/content/artifacts/adr/ADR-012-runbook-owns-incident-response.md b/docs/website/content/artifacts/adr/ADR-012-runbook-owns-incident-response.md new file mode 100644 index 00000000..6ebfe348 --- /dev/null +++ b/docs/website/content/artifacts/adr/ADR-012-runbook-owns-incident-response.md @@ -0,0 +1,97 @@ +--- +title: "ADR-012: Runbook owns incident response procedures; monitoring-setup owns detection only" +slug: ADR-012-runbook-owns-incident-response +weight: 270 +activity: "Design" +source: "02-design/adr/ADR-012-runbook-owns-incident-response.md" +generated: true +collection: adr +--- + +> **Source identity** (from `02-design/adr/ADR-012-runbook-owns-incident-response.md`): + +```yaml +ddx: + id: ADR-012 + depends_on: + - helix.prd +``` + +# ADR-012: Runbook owns incident response procedures; monitoring-setup owns detection only + +| Date | Status | Deciders | Related | Confidence | +|------|--------|----------|---------|------------| +| 2026-05-30 | Accepted | HELIX maintainers | plan-2026-05-30-artifact-types-and-concerns-audit, monitoring-setup, runbook | High | + +## Context + +| Aspect | Description | +|--------|-------------| +| Problem | The 2026-05-30 artifact-types-and-concerns audit flagged an ownership collision between `monitoring-setup` and `runbook`: both currently define incident-response routing. `monitoring-setup`'s template carries an `Incident Response` section (escalation paths, response procedures) that duplicates and competes with `runbook`'s `Common Incident Procedures` and escalation content. With two artifact types claiming the same surface, operators must reconcile two sources of truth at the moment they can least afford to. | +| Current State | `monitoring-setup/template.md` includes an `## Incident Response` section. `runbook/template.md` includes `## Common Incident Procedures` (with per-incident sections and a security/data-safety incident path) and explicit escalation routing. The two artifact types overlap on response procedures and escalation rather than partitioning detection from response. | +| Requirements | The catalog must assign incident-response ownership to exactly one artifact type. The other type must stop defining that surface so operators have a single canonical procedure document during an incident. | + +## Decision + +`monitoring-setup` owns **detection only**: SLI/SLO definitions, alert +routing inputs, threshold tuning, and the observability surface that +produces signals. + +`runbook` owns **incident response**: the procedures operators run when +those signals fire — recovery steps, common incident procedures, +escalation paths, and incident commander routing. + +In Phase 3, the `Incident Response` section is removed from the +`monitoring-setup` template. Any content from that section not already +covered moves to `runbook`. After Phase 3, the catalog validator can +treat presence of an `Incident Response` (or equivalent response-procedure) +H2 in `monitoring-setup` as a drift signal. + +**Key Points**: monitoring-setup = detection surface | runbook = operator +procedure | Phase 3 removes Incident Response from monitoring-setup +template | content not already in runbook moves there + +## Alternatives + +| Option | Pros | Cons | Evaluation | +|--------|------|------|------------| +| Keep both artifact types defining incident response; rely on authors to keep them in sync | No catalog change | The audit found they are already out of sync; two sources of truth during an incident is the worst possible time for ambiguity | Rejected: status quo the audit flagged | +| Move detection ownership into `runbook` and remove `monitoring-setup` | Single artifact type for the whole operational surface | Detection (SLI/SLO definitions, alert thresholds) and response (procedures, escalation) are authored by different people at different cadences; collapsing them loses that separation | Rejected: conflates two genuinely distinct authoring surfaces | +| **`monitoring-setup` owns detection; `runbook` owns response; Phase 3 removes the Incident Response section from monitoring-setup** | Clean partition aligned to the actual purpose of each artifact type; single source of truth for procedures; enforceable by validator | Requires a content migration for any monitoring-setup content not already in runbook | **Selected: smallest sufficient ownership fix** | + +## Consequences + +| Type | Impact | +|------|--------| +| Positive | Operators have a single canonical document (`runbook`) for incident procedures and escalation. | +| Positive | `monitoring-setup` becomes focused on its actual purpose — the observability surface — without bleeding into operator procedure. | +| Positive | The catalog validator can flag any future reintroduction of response procedures into `monitoring-setup` as drift. | +| Positive | Authoring cadences are no longer entangled: detection thresholds can evolve without re-touching response procedures, and vice versa. | +| Negative | Phase 3 must perform a content migration from `monitoring-setup`'s Incident Response section to `runbook` for anything not already covered. | +| Negative | Existing `monitoring-setup` artifacts authored under the previous contract need a one-time edit to remove the Incident Response section. | +| Neutral | The two artifact types continue to live side-by-side in the `05-deploy` activity; only their boundaries change. | + +## Risks + +| Risk | Prob | Impact | Mitigation | +|------|------|--------|------------| +| Content in `monitoring-setup`'s Incident Response section is lost rather than migrated | L | H | Phase 3 migration explicitly diffs the removed content against the runbook template before deleting; anything not already covered is added to runbook | +| Authors continue to add response procedures to `monitoring-setup` after Phase 3 | M | M | Catalog validator flags `Incident Response` (or equivalent response-procedure) H2 in `monitoring-setup` as drift | +| The detection/response partition is unclear at the boundary (e.g. alert routing) | M | M | Detection ends at "signal produced and routed to a destination"; response begins at "operator receives signal and acts." Alert routing destinations (PagerDuty, channel) are detection; what the recipient does is response | +| Existing references from outside the catalog point to the removed section | L | L | Phase 3 ships a redirect note in commit message and updates in-tree references in the same change | + +## Validation + +| Success Metric | Review Trigger | +|----------------|----------------| +| `monitoring-setup/template.md` contains no Incident Response section after Phase 3 | A PR reintroduces an Incident Response H2 (or equivalent response-procedure section) into `monitoring-setup` | +| All response-procedure content from the removed section is present in `runbook` (either pre-existing or migrated) | Phase 3 lands without a diff confirming migration coverage | +| The catalog validator flags `monitoring-setup` artifacts that carry response-procedure sections | A monitoring-setup artifact ships with response-procedure content and validation passes | +| Operators consulting `runbook` during an incident find escalation and procedure content without needing to cross-reference `monitoring-setup` | An incident retrospective surfaces split-source-of-truth as a contributing factor | + +## References + +- [Plan: artifact-types-and-concerns audit (2026-05-30)](/artifacts/plan-2026-05-30-artifact-types-and-concerns-audit/) +- [PRD](/artifacts/prd/) +- `workflows/activities/05-deploy/artifacts/monitoring-setup/template.md` +- `workflows/activities/05-deploy/artifacts/runbook/template.md` diff --git a/docs/website/content/artifacts/adr/ADR-013-cross-phase-informs-edges.md b/docs/website/content/artifacts/adr/ADR-013-cross-phase-informs-edges.md new file mode 100644 index 00000000..dac07c1c --- /dev/null +++ b/docs/website/content/artifacts/adr/ADR-013-cross-phase-informs-edges.md @@ -0,0 +1,91 @@ +--- +title: "ADR-013: Cross-phase informs edges are valid when declared on the downstream artifact" +slug: ADR-013-cross-phase-informs-edges +weight: 280 +activity: "Design" +source: "02-design/adr/ADR-013-cross-phase-informs-edges.md" +generated: true +collection: adr +--- + +> **Source identity** (from `02-design/adr/ADR-013-cross-phase-informs-edges.md`): + +```yaml +ddx: + id: ADR-013 + depends_on: + - helix.prd +``` + +# ADR-013: Cross-phase informs edges are valid when declared on the downstream artifact + +| Date | Status | Deciders | Related | Confidence | +|------|--------|----------|---------|------------| +| 2026-05-30 | Accepted | HELIX maintainers | artifact dependencies, schema validator | High | + +## Context + +| Aspect | Description | +|--------|-------------| +| Problem | The audit flagged the Deploy artifact `monitoring-setup` declaring `informs: metrics-dashboard`, an Iterate artifact in a later phase. It is unclear whether HELIX permits an artifact to declare an `informs` edge that crosses a phase boundary into a downstream phase, and whether the back-edge must also be declared on the consumer. | +| Current State | `workflows/activities/05-deploy/artifacts/monitoring-setup/meta.yml` declares `informs: metrics-dashboard`. `workflows/activities/06-iterate/artifacts/metrics-dashboard/meta.yml` declares both `informed_by` and `informs` lists but does not currently encode every back-edge for every upstream `informs`. The schema validator does not enforce edge symmetry across artifacts. | +| Requirements | The methodology must not forbid real evidence-flow edges that cross phases (production telemetry feeding dashboards is a real flow). At the same time, cross-phase edges must be explicit on both ends so that walking from either side reveals the relationship. | + +## Decision + +An artifact may declare `informs` an artifact in a later phase. The cross-phase +edge is valid when: + +1. The source artifact's purpose produces evidence the target consumes (the + edge encodes a real evidence flow, not a speculative association), and +2. The edge is declared on both sides — `informs` on the source artifact and + `informed_by` on the target artifact — or in a single canonical place + chosen by the consumer when only one direction can be authoritative. + +The `monitoring-setup` → `metrics-dashboard` edge is the canonical example: +production telemetry configured in Deploy feeds the dashboards consumed in +Iterate. The methodology accepts this as a valid cross-phase informs edge. + +**Key Points**: cross-phase informs allowed | edge must be real evidence flow | +declared on both ends (or canonical single place) | symmetry check is a +follow-up validator concern, not required by this ADR + +## Alternatives + +| Option | Pros | Cons | Evaluation | +|--------|------|------|------------| +| Forbid cross-phase informs edges entirely; restrict `informs` to same-phase artifacts | Simpler validator; clear phase boundary | Erases real evidence flows (Deploy telemetry → Iterate dashboards); forces artifacts to invent intermediate hops or drop the edge | Rejected: phase boundary is not a methodology firewall against evidence flow | +| Allow cross-phase informs but only when declared on the upstream side | Single source of truth | Walking from the consumer no longer reveals the upstream; reviewers must scan all earlier phases to find inbound edges | Rejected: breaks bidirectional discoverability | +| **Allow cross-phase informs; require declaration on both ends (or a canonical single place); leave symmetry enforcement as a follow-up validator concern** | Preserves real evidence flows; keeps edges discoverable from both ends; does not block this ADR on validator work | Phase 3 walk must check both sides; some artifacts will need back-edges added | **Selected: accepts the real flow without overcommitting the validator** | + +## Consequences + +| Type | Impact | +|------|--------| +| Positive | Real cross-phase evidence flows (e.g., Deploy telemetry → Iterate dashboards) are recognized as first-class edges. | +| Positive | Both-end declaration keeps the dependency graph walkable from either direction. | +| Positive | Phase 3 has a clear rule: walk all artifacts for `informs` edges and ensure the back-edge is declared. | +| Negative | Phase 3 must check edge symmetry across the full artifact set and add missing back-edges where the upstream declares `informs` but the downstream does not declare `informed_by`. | +| Neutral | The schema validator is unchanged in this ADR. Symmetry enforcement is a follow-up Phase 1B validator concern. | + +## Risks + +| Risk | Prob | Impact | Mitigation | +|------|------|--------|------------| +| Authors declare speculative cross-phase edges that are not real evidence flows | M | M | Phase 3 walk and reviewers reject edges that do not match the source artifact's stated purpose | +| Back-edges drift out of sync as artifacts evolve | M | M | Follow-up Phase 1B validator extension checks edge symmetry; until then, the Phase 3 walk establishes the baseline | +| Confusion about whether to declare on one end or both | L | L | This ADR states the rule: both ends, or a single canonical place chosen by the consumer when only one direction can be authoritative | + +## Validation + +| Success Metric | Review Trigger | +|----------------|----------------| +| Every `informs` edge declared on an upstream artifact has a matching `informed_by` entry on the downstream artifact (or a documented canonical-single-place choice) | A Phase 3 walk finds an upstream `informs` with no corresponding downstream `informed_by` | +| `monitoring-setup` → `metrics-dashboard` is preserved as a valid cross-phase informs edge | A review flags the edge as invalid because it crosses a phase boundary | +| Schema validator extension for edge symmetry is filed as a follow-up Phase 1B concern | Edge symmetry drift recurs and is not surfaced mechanically | + +## References + +- [PRD](/artifacts/prd/) +- [Plan: 2026-05-30 Artifact Types and Concerns Audit](/artifacts/plan-2026-05-30-artifact-types-and-concerns-audit/) +- ADR-004: Dependencies Encoding diff --git a/docs/website/content/artifacts/adr/ADR-014-architecture-style-slot-shape-deferred.md b/docs/website/content/artifacts/adr/ADR-014-architecture-style-slot-shape-deferred.md new file mode 100644 index 00000000..87d18b9f --- /dev/null +++ b/docs/website/content/artifacts/adr/ADR-014-architecture-style-slot-shape-deferred.md @@ -0,0 +1,89 @@ +--- +title: "ADR-014: Deferred: architecture-style slot occupants remain four separate concerns" +slug: ADR-014-architecture-style-slot-shape-deferred +weight: 290 +activity: "Design" +source: "02-design/adr/ADR-014-architecture-style-slot-shape-deferred.md" +generated: true +collection: adr +--- +# ADR-014: Deferred: architecture-style slot occupants remain four separate concerns + +| Date | Status | Deciders | Related | Confidence | +|------|--------|----------|---------|------------| +| 2026-05-30 | Deferred | HELIX maintainers | concerns/onion-architecture, concerns/clean-architecture, concerns/hexagonal-architecture, concerns/classic-layered, ADR-006 | Medium | + +## Context + +| Aspect | Description | +|--------|-------------| +| Problem | The architecture-style slot has four occupants (`onion-architecture`, `clean-architecture`, `hexagonal-architecture`, `classic-layered`) that sustain substantial content overlap by necessity — they are mutually exclusive at composition time and share the same shape (layering, dependency direction, boundary rules). The audit asked whether they should collapse into one concern with four variants, or remain as four separate concerns. | +| Current State | Each occupant ships its own `concern.md` + `practices.md`. The orthogonality audit classified the cluster as by-design slot competition. The verbosity attributed to these concerns is largely a consequence of the concern file shape (boundary restated in `concern.md`, `practices.md` "Boundary with neighbors", and Constraints/Drift Signals/Quality Gates), which is addressed separately by the concern file shape ADR. | +| Requirements | The audit needs a recorded decision so Phase 3 can proceed without re-opening this question. The decision must respect the audit's scope (it does not own the slot model itself). | + +## Decision + +We defer the question of whether the four architecture-style concerns collapse +into one concern with four variants. For this audit, the four concerns remain +separate — each keeps its own `concern.md` + `practices.md` and follows the +concern file shape ADR so the boundary is stated once. + +The question of per-occupant concern vs one concern with variants touches the +**slot model itself**, which is broader than this audit's scope. Collapsing +the architecture-style slot now would impose a shape on the other slot +families — language-runtime (`go-std` / `python-uv` / `rust-cargo` / `scala-sbt` +/ `typescript-bun`), e2e-framework (`e2e-kind` / `e2e-playwright`), and +demo-framework (`demo-asciinema` / `demo-playwright`) — without the +methodology-level analysis that decision warrants. That analysis is a separate +follow-up audit of the slot model, not a finding of the artifact-types-and- +concerns audit. + +The plan's Open Questions section preserves the question for that follow-up. + +**Key Points**: defer — not reject | four occupants remain separate concerns +for now | the slot-model decision is its own audit | concern file shape ADR +handles the restated-boundary verbosity independently + +## Alternatives + +| Option | Pros | Cons | Evaluation | +|--------|------|------|------------| +| Collapse the four architecture-style concerns into one concern with four variants | Removes the most-visible content overlap; one canonical boundary statement | Imposes a per-slot shape decision (variant container) on language-runtime, e2e-framework, and demo-framework slots without the broader methodology analysis; reshapes the slot model from inside one audit | Rejected for now: out of scope for this audit | +| Keep four separate concerns but rewrite each to be a thin delta against an "architecture-style overview" parent concern | Reduces restatement; preserves per-occupant addressability | Introduces a parent/child shape that the slot model doesn't currently support; same methodology-shape consequence as the collapse option | Rejected for now: same scope problem | +| **Keep the four concerns separate; defer the slot-model question to a dedicated follow-up audit** | Respects audit scope; lets the concern file shape ADR address the verbosity root cause without prejudging the slot model; preserves the question for the right venue | The visible content overlap persists until the slot-model audit lands | **Selected: scope-respecting deferral** | + +## Consequences + +| Type | Impact | +|------|--------| +| Positive | Phase 3 of the audit does not collapse these four concerns; each keeps its own `concern.md` + `practices.md`. | +| Positive | The slot-model decision is preserved for an audit that can analyze all four slot families coherently (architecture-style, language-runtime, e2e-framework, demo-framework). | +| Positive | The concern file shape ADR addresses the restated-boundary verbosity independently of the slot-model question, so the architecture-style concerns benefit from that fix without prejudging this one. | +| Negative | The visible content overlap among onion / clean / hexagonal / classic-layered persists until the slot-model audit lands. | +| Negative | Readers comparing the four occupants still navigate four separate files rather than one variant document. | +| Neutral | The architecture-style slot remains mutually exclusive at composition time; readers still pick exactly one. | + +## Risks + +| Risk | Prob | Impact | Mitigation | +|------|------|--------|------------| +| The deferred question is forgotten and the four concerns drift further apart | M | M | Plan's Open Questions section records the question; the follow-up slot-model audit picks it up | +| Editors trim restatement per-file in ways that re-introduce drift between the four occupants | M | M | The concern file shape ADR enforces "boundary stated once"; per-occupant trims reference-not-restate the slot's shared shape | +| The slot-model audit collapses the architecture-style slot later and invalidates per-occupant edits made in the meantime | L | M | Phase 3 edits to these four concerns are limited to file-shape compliance, not structural rewrites | + +## Validation + +| Success Metric | Review Trigger | +|----------------|----------------| +| The four architecture-style concerns each comply with the concern file shape ADR (boundary stated once) | A subsequent audit finds restated boundary prose across the four occupants | +| The plan's Open Questions section preserves the slot-model question for a follow-up audit | The question is silently dropped without an audit landing | +| Phase 3 does not include structural rewrites of these four concerns | A Phase 3 commit collapses or restructures the architecture-style slot | + +## References + +- Plan: docs/helix/02-design/plan-2026-05-30-artifact-types-and-concerns-audit.md (Open Questions item 7) +- ADR-006: Concern boundary lives once +- workflows/concerns/onion-architecture/ +- workflows/concerns/clean-architecture/ +- workflows/concerns/hexagonal-architecture/ +- workflows/concerns/classic-layered/ diff --git a/docs/website/content/artifacts/adr/_index.md b/docs/website/content/artifacts/adr/_index.md index 9c759070..aabff814 100644 --- a/docs/website/content/artifacts/adr/_index.md +++ b/docs/website/content/artifacts/adr/_index.md @@ -9,3 +9,14 @@ generated: true - [ADR-002: HELIX Tracker Write Safety Model](adr-002-tracker-write-safety-model/) - [ADR-003: Autonomy is a Three-Position Spectrum, Not a Fixed Level](adr-003-autonomy-spectrum/) +- [ADR-004: Artifact Dependencies Are Encoded in `meta.yml.relationships` Only (No Separate `dependencies.yaml`)](adr-004-dependencies-encoding/) +- [ADR-005: Concern practices.md is HELIX-activity-keyed](adr-005-concern-practices-activity-keyed/) +- [ADR-006: Concern boundary lives once, in concern.md](adr-006-concern-boundary-lives-once/) +- [ADR-007: Empty ADR References Headers Are Dropped; Required Only When a Concern Cites a Specific ADR](adr-007-empty-adr-references-headers-dropped/) +- [ADR-008: data-PRD Is a Kind-Switch Variant of PRD, Not a Sibling Artifact](adr-008-data-prd-is-prd-kind-variant/) +- [ADR-009: Acceptance Criteria Live in User-Stories; Feature-Specification Owns Functional Areas and Decomposition](adr-009-acceptance-criteria-ownership/) +- [ADR-010: feature-registry and parking-lot stay separate with an explicit handoff](adr-010-feature-registry-parking-lot-handoff/) +- [ADR-011: ux-radix owns Radix and shadcn prescriptions; react-nextjs references](adr-011-ux-radix-owns-shadcn-prescription/) +- [ADR-012: Runbook owns incident response procedures; monitoring-setup owns detection only](adr-012-runbook-owns-incident-response/) +- [ADR-013: Cross-phase informs edges are valid when declared on the downstream artifact](adr-013-cross-phase-informs-edges/) +- [ADR-014: Deferred: architecture-style slot occupants remain four separate concerns](adr-014-architecture-style-slot-shape-deferred/) diff --git a/docs/website/content/artifacts/architecture.md b/docs/website/content/artifacts/architecture.md index e57bf771..19e8041c 100644 --- a/docs/website/content/artifacts/architecture.md +++ b/docs/website/content/artifacts/architecture.md @@ -1,7 +1,7 @@ --- title: "Architecture" slug: architecture -weight: 190 +weight: 300 activity: "Design" source: "02-design/architecture.md" generated: true diff --git a/docs/website/content/artifacts/contracts/API-001-helix-tracker-mutation.md b/docs/website/content/artifacts/contracts/API-001-helix-tracker-mutation.md index 0ff56a8e..19eb58d3 100644 --- a/docs/website/content/artifacts/contracts/API-001-helix-tracker-mutation.md +++ b/docs/website/content/artifacts/contracts/API-001-helix-tracker-mutation.md @@ -1,7 +1,7 @@ --- title: "API Contract: HELIX Tracker Mutation Surface [FEAT-002]" slug: API-001-helix-tracker-mutation -weight: 200 +weight: 310 activity: "Design" source: "02-design/contracts/API-001-helix-tracker-mutation.md" generated: true diff --git a/docs/website/content/artifacts/contracts/CONTRACT-001-ddx-helix-boundary.md b/docs/website/content/artifacts/contracts/CONTRACT-001-ddx-helix-boundary.md index b2c35197..13812d6e 100644 --- a/docs/website/content/artifacts/contracts/CONTRACT-001-ddx-helix-boundary.md +++ b/docs/website/content/artifacts/contracts/CONTRACT-001-ddx-helix-boundary.md @@ -1,7 +1,7 @@ --- title: "CONTRACT-001: DDx / HELIX Boundary Contract" slug: CONTRACT-001-ddx-helix-boundary -weight: 210 +weight: 320 activity: "Design" source: "02-design/contracts/CONTRACT-001-ddx-helix-boundary.md" generated: true diff --git a/docs/website/content/artifacts/contracts/CONTRACT-002-helix-execution-doc-conventions.md b/docs/website/content/artifacts/contracts/CONTRACT-002-helix-execution-doc-conventions.md index 6f7e1b9c..5440fc05 100644 --- a/docs/website/content/artifacts/contracts/CONTRACT-002-helix-execution-doc-conventions.md +++ b/docs/website/content/artifacts/contracts/CONTRACT-002-helix-execution-doc-conventions.md @@ -1,7 +1,7 @@ --- title: "CONTRACT-002: HELIX Execution-Document Conventions" slug: CONTRACT-002-helix-execution-doc-conventions -weight: 220 +weight: 330 activity: "Design" source: "02-design/contracts/CONTRACT-002-helix-execution-doc-conventions.md" generated: true diff --git a/docs/website/content/artifacts/contracts/CONTRACT-003-ddx-adapter-boundary.md b/docs/website/content/artifacts/contracts/CONTRACT-003-ddx-adapter-boundary.md index 6a163432..cbdc8f38 100644 --- a/docs/website/content/artifacts/contracts/CONTRACT-003-ddx-adapter-boundary.md +++ b/docs/website/content/artifacts/contracts/CONTRACT-003-ddx-adapter-boundary.md @@ -1,7 +1,7 @@ --- title: "CONTRACT-003: DDx Adapter Boundary" slug: CONTRACT-003-ddx-adapter-boundary -weight: 230 +weight: 340 activity: "Design" source: "02-design/contracts/CONTRACT-003-ddx-adapter-boundary.md" generated: true diff --git a/docs/website/content/artifacts/contracts/README.md b/docs/website/content/artifacts/contracts/README.md index ec1c8305..adabd30a 100644 --- a/docs/website/content/artifacts/contracts/README.md +++ b/docs/website/content/artifacts/contracts/README.md @@ -1,7 +1,7 @@ --- title: "Contracts" slug: README -weight: 240 +weight: 350 activity: "Design" source: "02-design/contracts/README.md" generated: true diff --git a/docs/website/content/artifacts/data-design.md b/docs/website/content/artifacts/data-design.md index 08b9f699..424743f1 100644 --- a/docs/website/content/artifacts/data-design.md +++ b/docs/website/content/artifacts/data-design.md @@ -1,7 +1,7 @@ --- title: "Data Design — HELIX Bead Tracker" slug: data-design -weight: 250 +weight: 360 activity: "Design" source: "02-design/data-design.md" generated: true diff --git a/docs/website/content/artifacts/decisions/data-design-restoration-decision.md b/docs/website/content/artifacts/decisions/data-design-restoration-decision.md index c64ebdb9..0ecd459f 100644 --- a/docs/website/content/artifacts/decisions/data-design-restoration-decision.md +++ b/docs/website/content/artifacts/decisions/data-design-restoration-decision.md @@ -1,7 +1,7 @@ --- title: "Data Design — Restoration Decision" slug: data-design-restoration-decision -weight: 260 +weight: 370 activity: "Design" source: "02-design/decisions/data-design-restoration-decision.md" generated: true diff --git a/docs/website/content/artifacts/decisions/deployment-checklist-restoration-decision.md b/docs/website/content/artifacts/decisions/deployment-checklist-restoration-decision.md index 151a151e..1c35e697 100644 --- a/docs/website/content/artifacts/decisions/deployment-checklist-restoration-decision.md +++ b/docs/website/content/artifacts/decisions/deployment-checklist-restoration-decision.md @@ -1,7 +1,7 @@ --- title: "Deployment Checklist — Restoration Decision" slug: deployment-checklist-restoration-decision -weight: 600 +weight: 710 activity: "Deploy" source: "05-deploy/decisions/deployment-checklist-restoration-decision.md" generated: true diff --git a/docs/website/content/artifacts/decisions/improvement-backlog-restoration-decision.md b/docs/website/content/artifacts/decisions/improvement-backlog-restoration-decision.md index 5af94759..aa4d19e9 100644 --- a/docs/website/content/artifacts/decisions/improvement-backlog-restoration-decision.md +++ b/docs/website/content/artifacts/decisions/improvement-backlog-restoration-decision.md @@ -1,7 +1,7 @@ --- title: "Improvement Backlog — Restoration Decision" slug: improvement-backlog-restoration-decision -weight: 680 +weight: 790 activity: "Iterate" source: "06-iterate/decisions/improvement-backlog-restoration-decision.md" generated: true diff --git a/docs/website/content/artifacts/decisions/metrics-dashboard-restoration-decision.md b/docs/website/content/artifacts/decisions/metrics-dashboard-restoration-decision.md index 802e6bb9..88e8e2e6 100644 --- a/docs/website/content/artifacts/decisions/metrics-dashboard-restoration-decision.md +++ b/docs/website/content/artifacts/decisions/metrics-dashboard-restoration-decision.md @@ -1,7 +1,7 @@ --- title: "Metrics Dashboard — Restoration Decision" slug: metrics-dashboard-restoration-decision -weight: 690 +weight: 800 activity: "Iterate" source: "06-iterate/decisions/metrics-dashboard-restoration-decision.md" generated: true diff --git a/docs/website/content/artifacts/decisions/monitoring-setup-restoration-decision.md b/docs/website/content/artifacts/decisions/monitoring-setup-restoration-decision.md index ce23f0f6..1d044d2f 100644 --- a/docs/website/content/artifacts/decisions/monitoring-setup-restoration-decision.md +++ b/docs/website/content/artifacts/decisions/monitoring-setup-restoration-decision.md @@ -1,7 +1,7 @@ --- title: "Monitoring Setup — Restoration Decision" slug: monitoring-setup-restoration-decision -weight: 610 +weight: 720 activity: "Deploy" source: "05-deploy/decisions/monitoring-setup-restoration-decision.md" generated: true diff --git a/docs/website/content/artifacts/decisions/release-notes-restoration-decision.md b/docs/website/content/artifacts/decisions/release-notes-restoration-decision.md index 63f956cd..0012d610 100644 --- a/docs/website/content/artifacts/decisions/release-notes-restoration-decision.md +++ b/docs/website/content/artifacts/decisions/release-notes-restoration-decision.md @@ -1,7 +1,7 @@ --- title: "Release Notes — Restoration Decision" slug: release-notes-restoration-decision -weight: 620 +weight: 730 activity: "Deploy" source: "05-deploy/decisions/release-notes-restoration-decision.md" generated: true diff --git a/docs/website/content/artifacts/decisions/runbook-restoration-decision.md b/docs/website/content/artifacts/decisions/runbook-restoration-decision.md index 35fdb2e9..996fc11c 100644 --- a/docs/website/content/artifacts/decisions/runbook-restoration-decision.md +++ b/docs/website/content/artifacts/decisions/runbook-restoration-decision.md @@ -1,7 +1,7 @@ --- title: "Runbook — Restoration Decision" slug: runbook-restoration-decision -weight: 630 +weight: 740 activity: "Deploy" source: "05-deploy/decisions/runbook-restoration-decision.md" generated: true diff --git a/docs/website/content/artifacts/decisions/security-architecture-restoration-decision.md b/docs/website/content/artifacts/decisions/security-architecture-restoration-decision.md index 454397d6..baf4a830 100644 --- a/docs/website/content/artifacts/decisions/security-architecture-restoration-decision.md +++ b/docs/website/content/artifacts/decisions/security-architecture-restoration-decision.md @@ -1,7 +1,7 @@ --- title: "Security Architecture — Restoration Decision" slug: security-architecture-restoration-decision -weight: 270 +weight: 380 activity: "Design" source: "02-design/decisions/security-architecture-restoration-decision.md" generated: true diff --git a/docs/website/content/artifacts/decisions/security-metrics-restoration-decision.md b/docs/website/content/artifacts/decisions/security-metrics-restoration-decision.md index 509b16c5..7d7e5e42 100644 --- a/docs/website/content/artifacts/decisions/security-metrics-restoration-decision.md +++ b/docs/website/content/artifacts/decisions/security-metrics-restoration-decision.md @@ -1,7 +1,7 @@ --- title: "Security Metrics — Restoration Decision" slug: security-metrics-restoration-decision -weight: 700 +weight: 810 activity: "Iterate" source: "06-iterate/decisions/security-metrics-restoration-decision.md" generated: true diff --git a/docs/website/content/artifacts/deployment-checklist.md b/docs/website/content/artifacts/deployment-checklist.md index 44ef884e..77af3de8 100644 --- a/docs/website/content/artifacts/deployment-checklist.md +++ b/docs/website/content/artifacts/deployment-checklist.md @@ -1,7 +1,7 @@ --- title: "Deployment Checklist — HELIX Plugin and Website Release" slug: deployment-checklist -weight: 640 +weight: 750 activity: "Deploy" source: "05-deploy/deployment-checklist.md" generated: true diff --git a/docs/website/content/artifacts/executions/README.md b/docs/website/content/artifacts/executions/README.md index 9f79b41e..782c90ff 100644 --- a/docs/website/content/artifacts/executions/README.md +++ b/docs/website/content/artifacts/executions/README.md @@ -1,7 +1,7 @@ --- title: "Execution Docs" slug: README -weight: 570 +weight: 680 activity: "Test" source: "03-test/executions/README.md" generated: true diff --git a/docs/website/content/artifacts/hero-image-concepts/HERO-2026-05-12-document-spine-helix.md b/docs/website/content/artifacts/hero-image-concepts/HERO-2026-05-12-document-spine-helix.md index b00cf785..51bb5cd6 100644 --- a/docs/website/content/artifacts/hero-image-concepts/HERO-2026-05-12-document-spine-helix.md +++ b/docs/website/content/artifacts/hero-image-concepts/HERO-2026-05-12-document-spine-helix.md @@ -1,7 +1,7 @@ --- title: "HERO-2026-05-12 Document Spine Helix Concept" slug: HERO-2026-05-12-document-spine-helix -weight: 280 +weight: 390 activity: "Design" source: "02-design/hero-image-concepts/HERO-2026-05-12-document-spine-helix.md" generated: true diff --git a/docs/website/content/artifacts/homepage-worked-example-graph.md b/docs/website/content/artifacts/homepage-worked-example-graph.md index 006250ba..704fe5ee 100644 --- a/docs/website/content/artifacts/homepage-worked-example-graph.md +++ b/docs/website/content/artifacts/homepage-worked-example-graph.md @@ -1,7 +1,7 @@ --- title: "Homepage Worked Example Graph" slug: homepage-worked-example-graph -weight: 290 +weight: 400 activity: "Design" source: "02-design/homepage-worked-example-graph.md" generated: true diff --git a/docs/website/content/artifacts/improvement-backlog.md b/docs/website/content/artifacts/improvement-backlog.md index d10130e4..e105579a 100644 --- a/docs/website/content/artifacts/improvement-backlog.md +++ b/docs/website/content/artifacts/improvement-backlog.md @@ -1,7 +1,7 @@ --- title: "Improvement Backlog — HELIX 2026-Q2" slug: improvement-backlog -weight: 710 +weight: 820 activity: "Iterate" source: "06-iterate/improvement-backlog.md" generated: true diff --git a/docs/website/content/artifacts/metrics-dashboard.md b/docs/website/content/artifacts/metrics-dashboard.md index 849d3832..f145cd22 100644 --- a/docs/website/content/artifacts/metrics-dashboard.md +++ b/docs/website/content/artifacts/metrics-dashboard.md @@ -1,7 +1,7 @@ --- title: "Metrics Dashboard: HELIX 2026-Q2 (post-`v0.3.3`)" slug: metrics-dashboard -weight: 720 +weight: 830 activity: "Iterate" source: "06-iterate/metrics-dashboard.md" generated: true diff --git a/docs/website/content/artifacts/monitoring-setup.md b/docs/website/content/artifacts/monitoring-setup.md index 3c3d8a7b..43e56a13 100644 --- a/docs/website/content/artifacts/monitoring-setup.md +++ b/docs/website/content/artifacts/monitoring-setup.md @@ -1,7 +1,7 @@ --- title: "Monitoring Setup — `ddx-server`" slug: monitoring-setup -weight: 650 +weight: 760 activity: "Deploy" source: "05-deploy/monitoring-setup.md" generated: true diff --git a/docs/website/content/artifacts/plan-2026-05-17-claude-code-install-hardening.md b/docs/website/content/artifacts/plan-2026-05-17-claude-code-install-hardening.md index ae950f30..ac039aaa 100644 --- a/docs/website/content/artifacts/plan-2026-05-17-claude-code-install-hardening.md +++ b/docs/website/content/artifacts/plan-2026-05-17-claude-code-install-hardening.md @@ -1,7 +1,7 @@ --- title: "Plan: harden the Claude Code install surface" slug: plan-2026-05-17-claude-code-install-hardening -weight: 300 +weight: 410 activity: "Design" source: "02-design/plan-2026-05-17-claude-code-install-hardening.md" generated: true diff --git a/docs/website/content/artifacts/plan-2026-05-17-fold-datahelix-templates.md b/docs/website/content/artifacts/plan-2026-05-17-fold-datahelix-templates.md index bdd49641..ddc0cd57 100644 --- a/docs/website/content/artifacts/plan-2026-05-17-fold-datahelix-templates.md +++ b/docs/website/content/artifacts/plan-2026-05-17-fold-datahelix-templates.md @@ -1,7 +1,7 @@ --- title: "Plan: Fold DataHelix templates into the HELIX catalog" slug: plan-2026-05-17-fold-datahelix-templates -weight: 310 +weight: 420 activity: "Design" source: "02-design/plan-2026-05-17-fold-datahelix-templates.md" generated: true diff --git a/docs/website/content/artifacts/plan-2026-05-17-refresh-capability.md b/docs/website/content/artifacts/plan-2026-05-17-refresh-capability.md index e0c68396..262aa152 100644 --- a/docs/website/content/artifacts/plan-2026-05-17-refresh-capability.md +++ b/docs/website/content/artifacts/plan-2026-05-17-refresh-capability.md @@ -1,7 +1,7 @@ --- title: "Plan: §Refresh as a first-class HELIX skill behavior" slug: plan-2026-05-17-refresh-capability -weight: 320 +weight: 430 activity: "Design" source: "02-design/plan-2026-05-17-refresh-capability.md" generated: true diff --git a/docs/website/content/artifacts/plan-2026-05-24-helix-self-improvement.md b/docs/website/content/artifacts/plan-2026-05-24-helix-self-improvement.md index 011d81a3..6bcc5a5a 100644 --- a/docs/website/content/artifacts/plan-2026-05-24-helix-self-improvement.md +++ b/docs/website/content/artifacts/plan-2026-05-24-helix-self-improvement.md @@ -1,7 +1,7 @@ --- title: "Plan — HELIX self-improvement from the yolo-demo arc (2026-05-24, rev 2: codex-reviewed)" slug: plan-2026-05-24-helix-self-improvement -weight: 330 +weight: 440 activity: "Design" source: "02-design/plan-2026-05-24-helix-self-improvement.md" generated: true diff --git a/docs/website/content/artifacts/plan-2026-05-25-concern-slots-house-style.md b/docs/website/content/artifacts/plan-2026-05-25-concern-slots-house-style.md index c3ad7ba5..b1d13038 100644 --- a/docs/website/content/artifacts/plan-2026-05-25-concern-slots-house-style.md +++ b/docs/website/content/artifacts/plan-2026-05-25-concern-slots-house-style.md @@ -1,7 +1,7 @@ --- title: "Plan — concern slots + operator house-style (2026-05-25)" slug: plan-2026-05-25-concern-slots-house-style -weight: 340 +weight: 450 activity: "Design" source: "02-design/plan-2026-05-25-concern-slots-house-style.md" generated: true diff --git a/docs/website/content/artifacts/plan-2026-05-25-coverage-determinism.md b/docs/website/content/artifacts/plan-2026-05-25-coverage-determinism.md index fc5b7437..7b0e64cf 100644 --- a/docs/website/content/artifacts/plan-2026-05-25-coverage-determinism.md +++ b/docs/website/content/artifacts/plan-2026-05-25-coverage-determinism.md @@ -1,7 +1,7 @@ --- title: "Plan — decomposition/coverage determinism + e2e expectation (2026-05-25)" slug: plan-2026-05-25-coverage-determinism -weight: 350 +weight: 460 activity: "Design" source: "02-design/plan-2026-05-25-coverage-determinism.md" generated: true diff --git a/docs/website/content/artifacts/plan-2026-05-25-measurement-discipline.md b/docs/website/content/artifacts/plan-2026-05-25-measurement-discipline.md index b0e35bd1..bd3b2b80 100644 --- a/docs/website/content/artifacts/plan-2026-05-25-measurement-discipline.md +++ b/docs/website/content/artifacts/plan-2026-05-25-measurement-discipline.md @@ -1,7 +1,7 @@ --- title: "Plan — integrate the measurement-discipline learnings into HELIX (2026-05-25)" slug: plan-2026-05-25-measurement-discipline -weight: 360 +weight: 470 activity: "Design" source: "02-design/plan-2026-05-25-measurement-discipline.md" generated: true diff --git a/docs/website/content/artifacts/plan-2026-05-25-runtime-neutrality.md b/docs/website/content/artifacts/plan-2026-05-25-runtime-neutrality.md index 9f61d305..becf55e8 100644 --- a/docs/website/content/artifacts/plan-2026-05-25-runtime-neutrality.md +++ b/docs/website/content/artifacts/plan-2026-05-25-runtime-neutrality.md @@ -1,7 +1,7 @@ --- title: "Plan — DDx-reference runtime-neutrality remediation (2026-05-25)" slug: plan-2026-05-25-runtime-neutrality -weight: 370 +weight: 480 activity: "Design" source: "02-design/plan-2026-05-25-runtime-neutrality.md" generated: true diff --git a/docs/website/content/artifacts/plan-2026-05-25-sample-data.md b/docs/website/content/artifacts/plan-2026-05-25-sample-data.md index 0d1fdb45..63869a14 100644 --- a/docs/website/content/artifacts/plan-2026-05-25-sample-data.md +++ b/docs/website/content/artifacts/plan-2026-05-25-sample-data.md @@ -1,7 +1,7 @@ --- title: "Plan — sample-data concern (semantic faker, varied shapes) (2026-05-25)" slug: plan-2026-05-25-sample-data -weight: 380 +weight: 490 activity: "Design" source: "02-design/plan-2026-05-25-sample-data.md" generated: true diff --git a/docs/website/content/artifacts/plan-2026-05-25-ux-design.md b/docs/website/content/artifacts/plan-2026-05-25-ux-design.md index 0137fbe0..24aae577 100644 --- a/docs/website/content/artifacts/plan-2026-05-25-ux-design.md +++ b/docs/website/content/artifacts/plan-2026-05-25-ux-design.md @@ -1,7 +1,7 @@ --- title: "Plan — UX interface guidelines (ux-radix active-state) + DESIGN.md artifact (2026-05-25)" slug: plan-2026-05-25-ux-design -weight: 390 +weight: 500 activity: "Design" source: "02-design/plan-2026-05-25-ux-design.md" generated: true diff --git a/docs/website/content/artifacts/plan-2026-05-25-verification-concern.md b/docs/website/content/artifacts/plan-2026-05-25-verification-concern.md index dc19e922..5bdb9d9e 100644 --- a/docs/website/content/artifacts/plan-2026-05-25-verification-concern.md +++ b/docs/website/content/artifacts/plan-2026-05-25-verification-concern.md @@ -1,7 +1,7 @@ --- title: "Plan — aggressive-verification concern + AC-citation traceability (2026-05-25)" slug: plan-2026-05-25-verification-concern -weight: 400 +weight: 510 activity: "Design" source: "02-design/plan-2026-05-25-verification-concern.md" generated: true diff --git a/docs/website/content/artifacts/plan-2026-05-26-admin-console-auth-concerns.md b/docs/website/content/artifacts/plan-2026-05-26-admin-console-auth-concerns.md index 6ea2b08d..d77e2283 100644 --- a/docs/website/content/artifacts/plan-2026-05-26-admin-console-auth-concerns.md +++ b/docs/website/content/artifacts/plan-2026-05-26-admin-console-auth-concerns.md @@ -1,7 +1,7 @@ --- title: "Plan: `admin-console` + `auth` concerns — the foundational product surfaces (it.35)" slug: plan-2026-05-26-admin-console-auth-concerns -weight: 410 +weight: 520 activity: "Design" source: "02-design/plan-2026-05-26-admin-console-auth-concerns.md" generated: true diff --git a/docs/website/content/artifacts/plan-2026-05-26-bidirectional-traceability.md b/docs/website/content/artifacts/plan-2026-05-26-bidirectional-traceability.md index 371a8c18..e773f940 100644 --- a/docs/website/content/artifacts/plan-2026-05-26-bidirectional-traceability.md +++ b/docs/website/content/artifacts/plan-2026-05-26-bidirectional-traceability.md @@ -1,7 +1,7 @@ --- title: "Plan — it.36: bidirectional spec↔code traceability gate + spec-is-the-contract principle" slug: plan-2026-05-26-bidirectional-traceability -weight: 420 +weight: 530 activity: "Design" source: "02-design/plan-2026-05-26-bidirectional-traceability.md" generated: true diff --git a/docs/website/content/artifacts/plan-2026-05-26-complex-brief-robustness.md b/docs/website/content/artifacts/plan-2026-05-26-complex-brief-robustness.md index 087c54dc..e947a487 100644 --- a/docs/website/content/artifacts/plan-2026-05-26-complex-brief-robustness.md +++ b/docs/website/content/artifacts/plan-2026-05-26-complex-brief-robustness.md @@ -1,7 +1,7 @@ --- title: "Plan — complex-brief robustness: verification ENFORCEMENT + spike-first for unknowns (2026-05-26)" slug: plan-2026-05-26-complex-brief-robustness -weight: 430 +weight: 540 activity: "Design" source: "02-design/plan-2026-05-26-complex-brief-robustness.md" generated: true diff --git a/docs/website/content/artifacts/plan-2026-05-26-decomposition-boundaries.md b/docs/website/content/artifacts/plan-2026-05-26-decomposition-boundaries.md index e76218bb..1c79d6ff 100644 --- a/docs/website/content/artifacts/plan-2026-05-26-decomposition-boundaries.md +++ b/docs/website/content/artifacts/plan-2026-05-26-decomposition-boundaries.md @@ -1,7 +1,7 @@ --- title: "Plan — it.37: artifact-tier decomposition boundaries (PRD ↔ FEAT ↔ story)" slug: plan-2026-05-26-decomposition-boundaries -weight: 440 +weight: 550 activity: "Design" source: "02-design/plan-2026-05-26-decomposition-boundaries.md" generated: true diff --git a/docs/website/content/artifacts/plan-2026-05-26-spike-first-activation.md b/docs/website/content/artifacts/plan-2026-05-26-spike-first-activation.md index 929b0aa7..78f5d9eb 100644 --- a/docs/website/content/artifacts/plan-2026-05-26-spike-first-activation.md +++ b/docs/website/content/artifacts/plan-2026-05-26-spike-first-activation.md @@ -1,7 +1,7 @@ --- title: "Plan: spike-first ACTIVATION trigger (close the reframe-and-defer escape hatch)" slug: plan-2026-05-26-spike-first-activation -weight: 450 +weight: 560 activity: "Design" source: "02-design/plan-2026-05-26-spike-first-activation.md" generated: true diff --git a/docs/website/content/artifacts/plan-2026-05-26-spike-first-guidance-mandatory.md b/docs/website/content/artifacts/plan-2026-05-26-spike-first-guidance-mandatory.md index 1c740ba4..5860db89 100644 --- a/docs/website/content/artifacts/plan-2026-05-26-spike-first-guidance-mandatory.md +++ b/docs/website/content/artifacts/plan-2026-05-26-spike-first-guidance-mandatory.md @@ -1,7 +1,7 @@ --- title: "Plan: it.34 — \"defer the vendor\" is NOT a spike; a spike must EXERCISE the real provider and prove its economics (+ build the app-side data; unowned business calls → guidance-needed)" slug: plan-2026-05-26-spike-first-guidance-mandatory -weight: 460 +weight: 570 activity: "Design" source: "02-design/plan-2026-05-26-spike-first-guidance-mandatory.md" generated: true diff --git a/docs/website/content/artifacts/plan-2026-05-27-concern-expansion-and-converge.md b/docs/website/content/artifacts/plan-2026-05-27-concern-expansion-and-converge.md index e9e50e20..b1fd87e5 100644 --- a/docs/website/content/artifacts/plan-2026-05-27-concern-expansion-and-converge.md +++ b/docs/website/content/artifacts/plan-2026-05-27-concern-expansion-and-converge.md @@ -1,7 +1,7 @@ --- title: "Plan — concern-library expansion + make-concerns-bite + iterate-to-convergence (2026-05-27)" slug: plan-2026-05-27-concern-expansion-and-converge -weight: 470 +weight: 580 activity: "Design" source: "02-design/plan-2026-05-27-concern-expansion-and-converge.md" generated: true diff --git a/docs/website/content/artifacts/plan-2026-05-29-curated-page-scopes.md b/docs/website/content/artifacts/plan-2026-05-29-curated-page-scopes.md index 617fa8e7..9b605e1a 100644 --- a/docs/website/content/artifacts/plan-2026-05-29-curated-page-scopes.md +++ b/docs/website/content/artifacts/plan-2026-05-29-curated-page-scopes.md @@ -1,7 +1,7 @@ --- title: "Curated page scopes — lock the document boundaries before the prose pass" slug: plan-2026-05-29-curated-page-scopes -weight: 480 +weight: 590 activity: "Design" source: "02-design/plan-2026-05-29-curated-page-scopes.md" generated: true diff --git a/docs/website/content/artifacts/plan-2026-05-29-prose-quality-pass.md b/docs/website/content/artifacts/plan-2026-05-29-prose-quality-pass.md index 0526d6d1..eed5ba2f 100644 --- a/docs/website/content/artifacts/plan-2026-05-29-prose-quality-pass.md +++ b/docs/website/content/artifacts/plan-2026-05-29-prose-quality-pass.md @@ -1,7 +1,7 @@ --- title: "Prose-quality pass — evaluate and implement the expanded Vale findings" slug: plan-2026-05-29-prose-quality-pass -weight: 490 +weight: 600 activity: "Design" source: "02-design/plan-2026-05-29-prose-quality-pass.md" generated: true diff --git a/docs/website/content/artifacts/plan-2026-05-29-rename-authority-order.md b/docs/website/content/artifacts/plan-2026-05-29-rename-authority-order.md index 6f2d8864..ded2539a 100644 --- a/docs/website/content/artifacts/plan-2026-05-29-rename-authority-order.md +++ b/docs/website/content/artifacts/plan-2026-05-29-rename-authority-order.md @@ -1,7 +1,7 @@ --- title: "Rename \"authority order\" to \"authority hierarchy\"" slug: plan-2026-05-29-rename-authority-order -weight: 500 +weight: 610 activity: "Design" source: "02-design/plan-2026-05-29-rename-authority-order.md" generated: true diff --git a/docs/website/content/artifacts/plan-2026-05-30-artifact-types-and-concerns-audit.md b/docs/website/content/artifacts/plan-2026-05-30-artifact-types-and-concerns-audit.md index eae487b4..79f4c794 100644 --- a/docs/website/content/artifacts/plan-2026-05-30-artifact-types-and-concerns-audit.md +++ b/docs/website/content/artifacts/plan-2026-05-30-artifact-types-and-concerns-audit.md @@ -1,7 +1,7 @@ --- title: "Artifact-type and concern audit — gaps and remediation plan" slug: plan-2026-05-30-artifact-types-and-concerns-audit -weight: 510 +weight: 620 activity: "Design" source: "02-design/plan-2026-05-30-artifact-types-and-concerns-audit.md" generated: true diff --git a/docs/website/content/artifacts/prompt-iteration-protocol.md b/docs/website/content/artifacts/prompt-iteration-protocol.md index bb5ba1a6..1013324b 100644 --- a/docs/website/content/artifacts/prompt-iteration-protocol.md +++ b/docs/website/content/artifacts/prompt-iteration-protocol.md @@ -1,7 +1,7 @@ --- title: "HELIX Prompt Iteration Protocol and Evaluation Rubric" slug: prompt-iteration-protocol -weight: 730 +weight: 840 activity: "Iterate" source: "06-iterate/prompt-iteration-protocol.md" generated: true diff --git a/docs/website/content/artifacts/release-notes.md b/docs/website/content/artifacts/release-notes.md index c33446d7..686282bc 100644 --- a/docs/website/content/artifacts/release-notes.md +++ b/docs/website/content/artifacts/release-notes.md @@ -1,7 +1,7 @@ --- title: "Release Notes — HELIX v0.3.3" slug: release-notes -weight: 660 +weight: 770 activity: "Deploy" source: "05-deploy/release-notes.md" generated: true diff --git a/docs/website/content/artifacts/research-principles-injection-2026-04-05.md b/docs/website/content/artifacts/research-principles-injection-2026-04-05.md index d751d504..07c93cd2 100644 --- a/docs/website/content/artifacts/research-principles-injection-2026-04-05.md +++ b/docs/website/content/artifacts/research-principles-injection-2026-04-05.md @@ -1,7 +1,7 @@ --- title: "Research: Principles Injection Strategy Effectiveness" slug: research-principles-injection-2026-04-05 -weight: 740 +weight: 850 activity: "Iterate" source: "06-iterate/research-principles-injection-2026-04-05.md" generated: true diff --git a/docs/website/content/artifacts/runbook.md b/docs/website/content/artifacts/runbook.md index 8112aa7a..1d2459bb 100644 --- a/docs/website/content/artifacts/runbook.md +++ b/docs/website/content/artifacts/runbook.md @@ -1,7 +1,7 @@ --- title: "Runbook — `ddx-server`" slug: runbook -weight: 670 +weight: 780 activity: "Deploy" source: "05-deploy/runbook.md" generated: true diff --git a/docs/website/content/artifacts/security-architecture.md b/docs/website/content/artifacts/security-architecture.md index 93ac57df..f374ccaa 100644 --- a/docs/website/content/artifacts/security-architecture.md +++ b/docs/website/content/artifacts/security-architecture.md @@ -1,7 +1,7 @@ --- title: "Security Architecture — DDx Agent Execution Surface" slug: security-architecture -weight: 520 +weight: 630 activity: "Design" source: "02-design/security-architecture.md" generated: true diff --git a/docs/website/content/artifacts/security-metrics.md b/docs/website/content/artifacts/security-metrics.md index 340971b2..716e61d7 100644 --- a/docs/website/content/artifacts/security-metrics.md +++ b/docs/website/content/artifacts/security-metrics.md @@ -1,7 +1,7 @@ --- title: "Security Metrics — HELIX 2026-Q2 (post-`v0.3.3`)" slug: security-metrics -weight: 750 +weight: 860 activity: "Iterate" source: "06-iterate/security-metrics.md" generated: true diff --git a/docs/website/content/artifacts/solution-designs/SD-002-first-class-principles.md b/docs/website/content/artifacts/solution-designs/SD-002-first-class-principles.md index 0b1aa5a0..08b8f3e0 100644 --- a/docs/website/content/artifacts/solution-designs/SD-002-first-class-principles.md +++ b/docs/website/content/artifacts/solution-designs/SD-002-first-class-principles.md @@ -1,7 +1,7 @@ --- title: "Solution Design: SD-002 — First-Class Principles" slug: SD-002-first-class-principles -weight: 530 +weight: 640 activity: "Design" source: "02-design/solution-designs/SD-002-first-class-principles.md" generated: true diff --git a/docs/website/content/artifacts/technical-designs/TD-011-slider-autonomy-implementation.md b/docs/website/content/artifacts/technical-designs/TD-011-slider-autonomy-implementation.md index 9a2f1860..220f17ea 100644 --- a/docs/website/content/artifacts/technical-designs/TD-011-slider-autonomy-implementation.md +++ b/docs/website/content/artifacts/technical-designs/TD-011-slider-autonomy-implementation.md @@ -1,7 +1,7 @@ --- title: "TD-011: Slider Autonomy Implementation" slug: TD-011-slider-autonomy-implementation -weight: 540 +weight: 650 activity: "Design" source: "02-design/technical-designs/TD-011-slider-autonomy-implementation.md" generated: true diff --git a/docs/website/content/artifacts/technical-designs/TD-012-artifact-types-navigation.md b/docs/website/content/artifacts/technical-designs/TD-012-artifact-types-navigation.md index 91509167..eb556d37 100644 --- a/docs/website/content/artifacts/technical-designs/TD-012-artifact-types-navigation.md +++ b/docs/website/content/artifacts/technical-designs/TD-012-artifact-types-navigation.md @@ -1,7 +1,7 @@ --- title: "Technical Design: TD-012 — Artifact Types Navigation" slug: TD-012-artifact-types-navigation -weight: 550 +weight: 660 activity: "Design" source: "02-design/technical-designs/TD-012-artifact-types-navigation.md" generated: true diff --git a/docs/website/content/artifacts/technical-designs/TD-013-multi-runtime-install.md b/docs/website/content/artifacts/technical-designs/TD-013-multi-runtime-install.md index 6e366f39..7806e8f8 100644 --- a/docs/website/content/artifacts/technical-designs/TD-013-multi-runtime-install.md +++ b/docs/website/content/artifacts/technical-designs/TD-013-multi-runtime-install.md @@ -1,7 +1,7 @@ --- title: "Technical Design: TD-013 — Multi-Runtime Install" slug: TD-013-multi-runtime-install -weight: 560 +weight: 670 activity: "Design" source: "02-design/technical-designs/TD-013-multi-runtime-install.md" generated: true diff --git a/docs/website/content/artifacts/test-plans/TP-014-appendix-real-world-patterns.md b/docs/website/content/artifacts/test-plans/TP-014-appendix-real-world-patterns.md index ccf30402..5478b40c 100644 --- a/docs/website/content/artifacts/test-plans/TP-014-appendix-real-world-patterns.md +++ b/docs/website/content/artifacts/test-plans/TP-014-appendix-real-world-patterns.md @@ -1,7 +1,7 @@ --- title: "TP-014 Appendix A: Real-World Intent Patterns From Session Logs" slug: TP-014-appendix-real-world-patterns -weight: 580 +weight: 690 activity: "Test" source: "03-test/test-plans/TP-014-appendix-real-world-patterns.md" generated: true diff --git a/docs/website/content/artifacts/test-plans/TP-014-helix-workflow-coverage.md b/docs/website/content/artifacts/test-plans/TP-014-helix-workflow-coverage.md index c098266e..016355e5 100644 --- a/docs/website/content/artifacts/test-plans/TP-014-helix-workflow-coverage.md +++ b/docs/website/content/artifacts/test-plans/TP-014-helix-workflow-coverage.md @@ -1,7 +1,7 @@ --- title: "Workflow Test Plan: HELIX Across Five Runtimes" slug: TP-014-helix-workflow-coverage -weight: 590 +weight: 700 activity: "Test" source: "03-test/test-plans/TP-014-helix-workflow-coverage.md" generated: true diff --git a/docs/website/content/concerns/README-auth-family.md b/docs/website/content/concerns/README-auth-family.md new file mode 100644 index 00000000..57ef79e8 --- /dev/null +++ b/docs/website/content/concerns/README-auth-family.md @@ -0,0 +1,72 @@ +--- +title: "Auth family — ownership table" +generated: true +--- + +The auth family is a five-way cluster of concerns whose boundaries are easy to +restate and easy to confuse. Per ADR-006 (concern boundary lives once, in +`concern.md`), each concern in the family states its own boundary in its own +`concern.md`. This README is the **single ownership table** the family's +concerns cross-reference, so the per-concern files do not restate each other's +scope. + +Two cross-tree neighbors (`admin-console`, `unity-catalog`) are included +because they materially compose with the family but live outside it. + +## Family members and what each owns + +| Concern | Owns | +|---|---| +| `auth` | principal / session / account bootstrap; **session-token semantics** (issuance, rotation, revocation) | +| `authorization-model` | permission semantics (RBAC / ABAC / ReBAC) and the deny-by-default per-handler check | +| `multi-tenancy` | tenant predicate; tenant scoping rules (the inter-tenant guarantee) | +| `security-owasp` | hardening posture; **audit-logging policy** (what to log on authz denial) | + +The `auth-provider` slot (default filler: `auth-local-sessions`) sits beneath +`auth` and owns the *mechanism* (how identities/sessions are stored and +verified). A different slot filler (Auth0/OIDC, Clerk, …) is a swap, not a +rewrite. The slot is what `auth` defers to; it is not itself a member of the +ownership table because it has no scope `auth` does not already cover at the +surface level. + +## Neighbors that reference into the family + +| Neighbor | References | +|---|---| +| `admin-console` | `auth` (for operator-workflow gates — the operator surface is what `auth` gates) | +| `unity-catalog` | `authorization-model` (Databricks catalog grants compose with app-layer authz; neither substitutes for the other) | + +## Ordering invariants the family preserves + +1. **Authentication precedes authorization.** `auth` establishes the principal; + `authorization-model` decides what that principal may do. The two are + composable and ordered; an authz check on no principal is undefined. +2. **Tenant predicate precedes permission.** `multi-tenancy` guarantees the + record is in the caller's tenant; `authorization-model` then decides whether + the caller may act on it. A permission check on a wrong-tenant record is + still a cross-tenant leak. +3. **Hardening is a posture, not a substitute.** `security-owasp` owns the + umbrella (Broken Access Control, injection, CSRF, secret handling) plus + audit-logging policy on authz denial. It does not replace the per-handler + permission check (`authorization-model`) or the tenant predicate + (`multi-tenancy`). + +## Where ambiguous items land + +Two items historically fell through the gaps between these concerns; this +README records the owner so reviewers do not have to re-litigate: + +- **Session-token semantics** (issuance, rotation, revocation) → `auth`. + Tokens are the materialization of the session `auth` already owns. +- **Audit-logging policy** (what to log on authz denial, login failure, + privilege escalation) → `security-owasp`. Logging policy is a hardening + posture; the authz decision itself is `authorization-model`'s. + +## How to use this README + +- A concern in the family states **only its own** boundary in its `concern.md`. +- Where a concern would otherwise restate a neighbor's scope, it points here + ("see `workflows/concerns/README-auth-family.md` for the family ownership + table") and lets the table do the cross-reference work. +- A new concern that overlaps the family adds itself to the ownership table + in this README rather than restating the table in its own boundary. diff --git a/docs/website/content/concerns/a11y-wcag-aa.md b/docs/website/content/concerns/a11y-wcag-aa.md index fe2b44d5..6e204ba0 100644 --- a/docs/website/content/concerns/a11y-wcag-aa.md +++ b/docs/website/content/concerns/a11y-wcag-aa.md @@ -43,8 +43,6 @@ Selecting this concern requires these artifacts to change (a selected concern ab - DESIGN_SYSTEM: keyboard-navigable controls, AA contrast ratios, focus management, labeled inputs - TEST_PLAN: axe-core automated checks + manual screen-reader passes -## ADR References - ## Practices by activity Agents working in any of these activities inherit the practices below via the bead's context digest. diff --git a/docs/website/content/concerns/admin-console.md b/docs/website/content/concerns/admin-console.md index f8c10543..cb531087 100644 --- a/docs/website/content/concerns/admin-console.md +++ b/docs/website/content/concerns/admin-console.md @@ -20,14 +20,14 @@ ui, api This concern owns the **operator product surface and its workflow-coverage target**: for an operator-facing product, the human operator can do their -jobs-to-be-done from the UI. It is composable and stays out of four neighbours: +jobs-to-be-done from the UI. It is composable. -- **`verification`** owns *evidence* (work isn't done without an observed run). -- **`e2e-framework`** (slot) owns the *tool/form* of the e2e (browser runner, or - live-server HTTP for server-rendered — `verification` decides which). -- **`ux-radix`** owns *interaction quality* (active-state, focus, states). -- **`frontend-framework`** (slot) owns the *stack*; `security-owasp` the - hardening; `auth` gates access. +For access gating, `admin-console` references `auth` — the operator surface is +what `auth` gates; see [README-auth-family.md](../readme-auth-family/) for +the auth family ownership table. For evidence (`verification`), the e2e +tool/form (`e2e-framework` slot), interaction quality (`ux-radix`), the stack +(`frontend-framework` slot), and hardening (`security-owasp`), see those +concerns directly. `admin-console` owns the one thing those do not state: **the operator's jobs-to-be-done — CRUD over the core domain objects plus the domain's @@ -116,9 +116,11 @@ Agents working in any of these activities inherit the practices below via the be These practices realize the `admin-console` concern: the operator's jobs-to-be-done built as usable UI, with the primary operator workflow exercised -end-to-end through that UI. They do not restate evidence rules (`verification`), -the e2e tool (`e2e-framework`), interaction quality (`ux-radix`), the stack -(`frontend-framework`), hardening (`security-owasp`), or access gating (`auth`). +end-to-end through that UI. Access gating defers to `auth` — see +[README-auth-family.md](../readme-auth-family/) for the auth family ownership +table. Evidence rules (`verification`), the e2e tool (`e2e-framework`), +interaction quality (`ux-radix`), the stack (`frontend-framework`), and +hardening (`security-owasp`) each live in their own concerns. ## Design diff --git a/docs/website/content/concerns/auth-local-sessions.md b/docs/website/content/concerns/auth-local-sessions.md index 6d9ba6c2..4eaa54be 100644 --- a/docs/website/content/concerns/auth-local-sessions.md +++ b/docs/website/content/concerns/auth-local-sessions.md @@ -23,11 +23,13 @@ auth-provider This concern is the **default `auth-provider` slot filler** — a real, working, self-contained authentication backend that needs no external identity service. -It supplies the *mechanism* the `auth` concern's product surface sits on. It -does not define the surface (signup/login/roles UX — that is `auth`) and does -not own security hardening (that is `security-owasp`). An external IdP -(Auth0/OIDC, Clerk, …) is a **different filler of the same slot**, swapped in -without changing call sites. +It supplies the *mechanism* the `auth` concern's product surface sits on. An +external IdP (Auth0/OIDC, Clerk, …) is a **different filler of the same slot**, +swapped in without changing call sites. + +For the family ownership table (and the boundaries with `auth`, +`authorization-model`, `multi-tenancy`, and `security-owasp`) see +[README-auth-family.md](../readme-auth-family/). ## Components @@ -80,14 +82,20 @@ Agents working in any of these activities inherit the practices below via the be These practices realize the default `auth-provider` filler — a working local auth backend behind the provider interface the `auth` concern requires. They -cover the *mechanism*; the surface/roles/isolation requirements are in `auth`, -and hardening depth is in `security-owasp`. +cover the *mechanism*; for the family ownership table (auth / +authorization-model / multi-tenancy / security-owasp) see +[README-auth-family.md](../readme-auth-family/). -## Implementation +## Design - **Provider interface**: expose one auth/identity interface (`signup`, `authenticate`, `resolve_principal`, `logout`) that the app calls; keep local-session internals behind it. +- **Swap path**: document how an external IdP filler (Auth0/OIDC) replaces this + one — same interface, config-selected, no call-site change. + +## Build + - **Passwords**: hash with a salted, work-factored algorithm (PBKDF2/bcrypt/ argon2); never store or log plaintext; verify in constant time. - **Sessions**: issue a server-side session on login, referenced by an HttpOnly @@ -95,10 +103,8 @@ and hardening depth is in `security-owasp`. logout. - **Principal resolution**: `resolve_principal` returns the authenticated principal (+ account/tenant + role) for the request, or unauthenticated. -- **Swap path**: document how an external IdP filler (Auth0/OIDC) replaces this - one — same interface, config-selected, no call-site change. -## Quality Gates +## Test - Passwords are hashed (salted + work-factored); no plaintext anywhere. - Session cookie is HttpOnly; sessions validated server-side; logout clears them. diff --git a/docs/website/content/concerns/auth.md b/docs/website/content/concerns/auth.md index c3a118af..77503614 100644 --- a/docs/website/content/concerns/auth.md +++ b/docs/website/content/concerns/auth.md @@ -20,21 +20,17 @@ api, data, ui This concern owns the **authentication & account product surface** — that an account/multi-tenant product can actually be signed into and is scoped to its -principals. It is composable and does NOT fill a slot. It defers the *backend* -to the `auth-provider` slot, and stays out of three neighbours: - -- **`auth-provider`** (the slot, default `auth-local-sessions`) owns the - *mechanism* — how identities/sessions are stored and verified, and which - provider (local sessions, Auth0/OIDC, Clerk, …) is wired in. `auth` requires - the surface; the slot filler supplies the backend. -- **`security-owasp`** owns *security hardening* — injection, CSRF, secret - handling, password storage strength, rate limiting. `auth` requires that - isolation and roles are enforced; `security-owasp` governs how safely. -- **`admin-console`** owns the *operator workflows*; `auth` is what gates them. - -`auth` owns the one thing those do not state: **an account/tenant product is -not done until a real principal can sign up, sign in, and act only within their -authorized scope.** +principals, plus **session-token semantics** (issuance, rotation, revocation). +It is composable and does NOT fill a slot; it defers the *backend* to the +`auth-provider` slot (default `auth-local-sessions`). + +For the family ownership table (auth / authorization-model / multi-tenancy / +security-owasp, plus the admin-console and unity-catalog neighbors) see +[README-auth-family.md](../readme-auth-family/). + +`auth` owns the one thing the rest of the family does not state: **an +account/tenant product is not done until a real principal can sign up, sign in, +and act only within their authorized scope.** ## Components @@ -117,9 +113,10 @@ Agents working in any of these activities inherit the practices below via the be These practices realize the `auth` concern: a real, usable authentication & account surface, with authorization and isolation enforced through the -authenticated principal, and the backend behind the `auth-provider` slot. They -do not restate the provider mechanism (that is the slot filler's, e.g. -`auth-local-sessions`) or security hardening (that is `security-owasp`). +authenticated principal, and the backend behind the `auth-provider` slot. For +the family ownership table (and what `security-owasp`, `authorization-model`, +`multi-tenancy`, and the `auth-provider` slot own instead of `auth`) see +[README-auth-family.md](../readme-auth-family/). ## Design diff --git a/docs/website/content/concerns/authorization-model.md b/docs/website/content/concerns/authorization-model.md index 7288b9f1..326f630a 100644 --- a/docs/website/content/concerns/authorization-model.md +++ b/docs/website/content/concerns/authorization-model.md @@ -33,39 +33,19 @@ This concern owns the **permission model and its enforcement** — *what* an authenticated principal may do to the resources they can legitimately reach, and the discipline that every state-changing and data-returning handler asks that question and refuses by default. It is composable and does **not** fill a slot. -Three neighbours stay distinct, and this concern must not duplicate them: - -- **`auth`** owns **who you are** — signup, login, sessions, and that a real - principal exists and is scoped to its account. `auth` (and the `auth-provider` - slot behind it) answers *authentication*: "are you signed in, and as whom?". - This concern answers *authorization*: "given who you are, **may** you do this - to this resource?". `auth` already names "Authorization (RBAC)" and - "enforced server-side" as a requirement of its surface; this concern owns the - **model behind that requirement** and makes the per-handler permission check - reviewer-checkable. Do not restate signup/session lifecycle here; consume the - authenticated principal `auth` establishes. -- **`multi-tenancy`** owns **isolation between tenants** — that a record is even - *in the caller's tenant at all* (an *inter-tenant* guarantee). This concern - owns **intra-tenant permissions** — given a record the caller can legitimately - reach, *may they do this to it* ("may this member delete this project in - *their own* tenant?"). The two compose and are ordered: **the tenant predicate - must hold before a permission check is meaningful** — a correct permission - check on a record that belongs to the wrong tenant is still a cross-tenant - leak, so tenant scoping precedes authorization. `multi-tenancy` states this - reciprocally; do not fold tenant isolation into this concern, and do not let a - permission check stand in for the tenant predicate. -- **`security-owasp`** owns general hardening — injection, CSRF, secret - handling, input validation, TLS. **Broken Access Control is the OWASP umbrella - failure; this concern is the access-control *model* that prevents it.** - `security-owasp` says "authorization checked on every protected endpoint"; - this concern owns *how the decision is modeled and enforced* (deny-by-default, - a chosen model, a central decision point) so that the umbrella requirement has - a concrete, reviewer-checkable shape. - -This concern owns the one thing those do not state: **there is a deliberately -chosen permission model, every state-changing and data-returning handler -authorizes the principal against it deny-by-default, and a negative test proves -an unauthorized principal is refused.** + +For the family ownership table (auth / authorization-model / multi-tenancy / +security-owasp, plus the admin-console and unity-catalog neighbors), and the +ordering invariants (tenant predicate precedes permission; authn precedes +authz; hardening does not substitute for the model), see +[README-auth-family.md](../readme-auth-family/). + +This concern owns the one thing the rest of the family does not state: **there +is a deliberately chosen permission model, every state-changing and +data-returning handler authorizes the principal against it deny-by-default, and +a negative test proves an unauthorized principal is refused.** Broken Access +Control is the OWASP umbrella `security-owasp` names; this concern is the +access-control *model* that prevents it. ## Components @@ -222,13 +202,12 @@ Agents working in any of these activities inherit the practices below via the be These practices govern the **permission model and its enforcement** — *what* an authenticated principal may do, and the discipline that every privileged handler -asks and refuses by default. They sit alongside `auth` (who you are), -`multi-tenancy` (is the record even in your tenant — the predicate that precedes -any permission check), and `security-owasp` (Broken Access Control is the OWASP -umbrella this model prevents). Their one job is to make the **missing / -mis-placed authorization check** unreachable and **reviewer-checkable**. Each -MUST/SHOULD below is written so a reviewer can confirm or refute it against the -diff and the running system. +asks and refuses by default. For the family ownership table (auth / +authorization-model / multi-tenancy / security-owasp) see +[README-auth-family.md](../readme-auth-family/). Their one job is to make the +**missing / mis-placed authorization check** unreachable and +**reviewer-checkable**. Each MUST/SHOULD below is written so a reviewer can +confirm or refute it against the diff and the running system. ## Choose and record the model (Frame / Design) @@ -304,17 +283,6 @@ diff and the running system. (succeeds) and the denied (refused) cases **SHOULD** be exercised, so the guard branch is shown to exist and not be vacuously open. -## Boundary with neighbors - -- For **who the principal is** (signup, login, sessions, account bootstrap) - defer to `auth`; do not restate it here. This concern consumes the principal - and decides what it may do. -- For **is the record even in the caller's tenant** defer to `multi-tenancy`; - that predicate precedes — and never replaces — the permission check here. -- For general hardening (injection, CSRF, secrets, TLS) defer to - `security-owasp`; missing/broken authorization is its Broken Access Control - case, given a concrete model and per-handler discipline here. - ## Quality Gates - Permission **model recorded in an ADR** (RBAC / ABAC / ReBAC or a named diff --git a/docs/website/content/concerns/caching-strategy.md b/docs/website/content/concerns/caching-strategy.md index 320650e9..9da5f9e0 100644 --- a/docs/website/content/concerns/caching-strategy.md +++ b/docs/website/content/concerns/caching-strategy.md @@ -211,7 +211,7 @@ failure-handling policy a cache may participate in (`resilience`), or the schema of the store the cache fronts (`relational-data-modeling` / the `datastore` slot). A cache is always a derived, disposable copy; the origin is the truth. -## Cache only to hit a stated target +## Discover - Add a cache **only** to meet a named **performance NFR** against a **real read-heavy hot path or an expensive computation**. Record *which* path / @@ -221,7 +221,7 @@ slot). A cache is always a derived, disposable copy; the origin is the truth. - Before caching an expensive computation, confirm the cost is real (measured), not assumed; a material uncertainty is a `tech-spike`, not a silent cache. -## Pick the pattern from the consistency need +## Frame - Choose the read/write pattern deliberately and record **why** in the ADR: - **cache-aside (lazy)** / **read-through** for read-heavy data tolerant of @@ -234,8 +234,10 @@ slot). A cache is always a derived, disposable copy; the origin is the truth. accepted**. - Record the **consistency trade-off** the chosen pattern implies against the data's tolerance for staleness — never stumble into write-behind's window. +- Record in the ADR **what is deliberately not cached** (and why it must stay + fresh). -## State invalidation, TTL, and a staleness budget for every cache +## Design - Each cached dataset has an explicit **invalidation policy** — a TTL, explicit invalidation on write, or both — and a **named staleness budget** (the maximum @@ -244,9 +246,6 @@ slot). A cache is always a derived, disposable copy; the origin is the truth. - When using explicit invalidation, identify **every write path** that affects a cached key and invalidate/update on each; a missed write path is a stale-read bug. Record the cache **keys** and their TTLs in the technical-design. - -## Do not cache correctness-sensitive reads behind a stale copy - - Data where a stale read causes an **incorrect decision or side effect** — authorization/permission checks, balances or quotas that gate an action, anything needing **read-your-write** — is **not** served from a cache that can @@ -255,7 +254,7 @@ slot). A cache is always a derived, disposable copy; the origin is the truth. - The cache is **never the system of record**: nothing of record lives only in the cache; on a miss, eviction, or cache failure the **origin is the truth**. -## Protect hot keys from stampede +## Build - For each hot key, ensure expiry or cold start does **not** dogpile the origin: apply **single-flight / request coalescing** (one loader, others wait), @@ -265,17 +264,7 @@ slot). A cache is always a derived, disposable copy; the origin is the truth. is desired, record that as a `resilience` degradation decision — the cache composes as a fallback, it is not itself the resilience mechanism. -## Boundary with neighbors - -- The **performance NFR** owns the target; this concern owns the cache that - serves it. Do not restate the latency/throughput number — point at the NFR. -- **`resilience`** owns timeouts/retries/breakers/fallbacks and whether - stale-serve-on-failure is an accepted degradation; this concern owns the - cache's read/write/invalidation behavior. Caching ≠ resilience. -- **`relational-data-modeling`** / the **`datastore`** slot owns the system of - record the cache fronts; the cache holds a derived, disposable copy. - -## Quality Gates +## Test - Every cache **traces to a performance NFR and a real read-heavy hot path or expensive computation**; no cache exists without a target it serves (no @@ -294,3 +283,13 @@ slot). A cache is always a derived, disposable copy; the origin is the truth. TTL, or negative cache); a hot-key expiry does not dogpile the origin. - **What is deliberately not cached** (and why it must stay fresh) is recorded in the ADR. + +## Cross-cutting + +### Boundary with neighbors + +See `concern.md` for the canonical Boundary (vs the performance NFR, +`resilience`, `relational-data-modeling` / `datastore`). The cache is a +performance/staleness mechanism — defer failure-handling policy to +`resilience` and the schema of the system of record to the data-modeling / +store neighbors. diff --git a/docs/website/content/concerns/classic-layered.md b/docs/website/content/concerns/classic-layered.md index 2ca2e8ce..47658256 100644 --- a/docs/website/content/concerns/classic-layered.md +++ b/docs/website/content/concerns/classic-layered.md @@ -169,22 +169,45 @@ depend directly on the data-access layer.** These practices do not require an inverted persistence boundary — if review finds the team wants one, the `architecture-style` selection is wrong and a sibling should fill the slot. -## Layering and dependency direction +## Discover + +- Confirm the product does **not** require a swappable datastore, an + isolated-testable domain, or multiple symmetric driving adapters — if it + does, the `architecture-style` is mis-selected; recommend a + dependency-inversion sibling (`onion` / `hexagonal` / `clean`). +- Per KISS/YAGNI, do NOT introduce repository interfaces, ports, or a DI + container speculatively under this concern — that ceremony belongs to a + sibling style that was deliberately selected. + +## Frame + +- If a relaxed/open-layer policy is adopted (a layer may call any lower layer), + it MUST be recorded as a deliberate decision; even then, a layer MUST NOT + call a layer **above** it. +- If the product actually needs a **swappable datastore**, an + **infrastructure-free testable domain**, or **multiple symmetric driving + adapters**, the reviewer SHOULD flag that the `architecture-style` selection + is mismatched and recommend a dependency-inversion sibling, rather than + bolting partial inversion onto a classic-layered codebase. + +## Design - The code MUST be organized into recognizable layers — **presentation → application/service → domain → data-access** — with a discoverable mapping from layer to package/module/directory. - Each layer MUST depend **only on the layer directly beneath it** (closed-layer - default). If a relaxed/open-layer policy is adopted (a layer may call any - lower layer), it MUST be recorded as a deliberate decision; even then, a layer - MUST NOT call a layer **above** it. + default). - A lower layer MUST NOT import, reference, or depend on a higher layer (verify the import graph: the data-access layer has **zero** edges to the presentation or application layers). - The presentation layer MUST NOT call the data-access layer directly or touch persistence by other means; it MUST go through the application/domain layer. +- This style does NOT require persistence interfaces to be declared in the + domain layer, and does NOT require a composition root. Reviewers MUST NOT + flag the absence of an inverted persistence boundary as a defect under this + concern — that absence is the point. -## Keep behavior in its layer +## Build - Business rules MUST live in the domain/application layer, not in presentation controllers and not in the data-access layer (no business logic in SQL, @@ -193,31 +216,7 @@ inverted persistence boundary — if review finds the team wants one, the DAOs) and contain no business decisions; the presentation layer SHOULD contain no business decisions either. -## Accept the trade-off explicitly (do not pretend to be a sibling) - -- This style does NOT require persistence interfaces to be declared in the - domain layer, and does NOT require a composition root. Reviewers MUST NOT - flag the absence of an inverted persistence boundary as a defect under this - concern — that absence is the point. -- If the product actually needs a **swappable datastore**, an - **infrastructure-free testable domain**, or **multiple symmetric driving - adapters**, the reviewer SHOULD flag that the `architecture-style` selection - is mismatched and recommend a dependency-inversion sibling, rather than - bolting partial inversion onto a classic-layered codebase. -- Per KISS/YAGNI, do NOT introduce repository interfaces, ports, or a DI - container speculatively under this concern — that ceremony belongs to a - sibling style that was deliberately selected. - -## Boundary with sibling concerns - -- The **contents** of the domain layer (aggregates, invariants, ubiquitous - language) are governed by `domain-driven-design`, not here. Verify the model - sits in the domain layer; do not restate DDD modeling rules. -- Object-level collaboration patterns inside a layer are `design-patterns-gof`; - between-system integration is `enterprise-integration-patterns`. - Classic-layered only governs the macro tier-stacking across the codebase. - -## Quality Gates +## Test - Import-graph check: dependencies point **downward only** — no layer imports a layer above it; the data-access layer has zero edges to presentation or @@ -234,3 +233,14 @@ inverted persistence boundary — if review finds the team wants one, the an isolated-testable domain, or multiple symmetric driving adapters — if it does, the `architecture-style` is mis-selected; recommend a dependency-inversion sibling (`onion` / `hexagonal` / `clean`). + +## Cross-cutting + +### Boundary with sibling concerns + +- The **contents** of the domain layer (aggregates, invariants, ubiquitous + language) are governed by `domain-driven-design`, not here. Verify the model + sits in the domain layer; do not restate DDD modeling rules. +- Object-level collaboration patterns inside a layer are `design-patterns-gof`; + between-system integration is `enterprise-integration-patterns`. + Classic-layered only governs the macro tier-stacking across the codebase. diff --git a/docs/website/content/concerns/concurrency-model.md b/docs/website/content/concerns/concurrency-model.md index 6206e2f2..663be44d 100644 --- a/docs/website/content/concerns/concurrency-model.md +++ b/docs/website/content/concerns/concurrency-model.md @@ -483,25 +483,11 @@ reference those concerns at the seam and stay on the execution model. ## Boundary with neighbors -- **vs `enterprise-integration-patterns`**: the **channel/queue a worker - consumes** across a system boundary — its delivery guarantee, DLQ, router, - correlation id, wire shape — is EIP. The **worker pool's concurrency, the - job's idempotency, and its in-process failure visibility** are here. Idempotency - is shared vocabulary (EIP's Idempotent Receiver vs an idempotent job), split by - cause: EIP because the *channel* redelivers, this concern because the *worker* - retries. Do not restate channel/DLQ rules here. -- **vs `resilience`**: a **bounded work pool / semaphore / bounded channel that - caps in-flight work in steady state** is here (the execution model's native - bound). A **bulkhead** (per-dependency pool isolation), **load-shedding** under - overload, the **circuit breaker** and **timeout** are `resilience` (failure - containment). Backpressure as the steady-state shape of the execution model is - here; load-shedding as a failure response is resilience. Do not restate - breaker/bulkhead/shed machinery here. -- **vs `deployment-topology`**: **in-process** execution (threads / event loop / - goroutines / worker pool inside one deployable) is here; **process-level - scaling** (replica count, monolith-vs-microservices) is deployment-topology. - A horizontally-scaled fleet still needs an in-process model per replica — that - per-replica model is here. +See `concern.md` for the canonical Boundary (vs `enterprise-integration-patterns`, +`resilience`, `deployment-topology`). These practices stay on the in-process +execution model; reach to the neighbor named there for channel/DLQ rules +(EIP), bulkhead/breaker/load-shedding (`resilience`), or replica/deployable +count (`deployment-topology`). ## Quality Gates diff --git a/docs/website/content/concerns/cqrs.md b/docs/website/content/concerns/cqrs.md index 13574398..eb2aec0f 100644 --- a/docs/website/content/concerns/cqrs.md +++ b/docs/website/content/concerns/cqrs.md @@ -275,7 +275,7 @@ aggregate, defer to DDD; when it references the event store / projections as the source of truth, defer to event sourcing. Apply these **per bounded context** that selected CQRS — not to every context in the product. -## Decide the bounded context and consistency model before applying +## Discover - Confirm the context **earns** CQRS: a **collaborative** domain (many concurrent writers), a **task-based UI** (intents map to commands), **divergent read vs. @@ -283,18 +283,25 @@ that selected CQRS — not to every context in the product. neither side. If it is simple CRUD read the same way it is written, do **NOT** apply CQRS — use a single model (record the decision; Fowler: CQRS adds risky complexity, it is difficult to use well). + +## Frame + - Choose and **record the consistency model in an ADR**: **synchronous** (read model updated in the write transaction — strongly consistent, simpler, same store) vs. **eventually consistent** (read model updated out-of-band, separate store — scalable, but a stale-read window). Name the **bounded context**, the **decisive signal**, and (if eventually consistent) the **synchronization path** and the **staleness window** consumers must tolerate. +- A **synchronous** consistency model (read model updated in the write + transaction, same store) is the simpler choice when strong read-after-write is + required; prefer it unless the asymmetry/scale signal justifies the + eventually-consistent separation. Whichever is chosen MUST match the ADR. - The selection MUST drive the artifacts: the **technical-design** SHOULD show separate command/query models, the projection/read-model, and the eventual-consistency handling; the **data-design** SHOULD show the read schema distinct from the write schema and how they stay in sync. -## Separate the write model from the read model +## Design - The model handling **commands** (state changes) MUST be **distinct** from the model serving **queries** (reads) within the CQRS-selected context — not one @@ -304,9 +311,6 @@ that selected CQRS — not to every context in the product. method that mutates *and* returns the mutated view conflates the sides. - Queries MUST read from the **read model / projection**, NOT by loading write-model aggregates to read fields off them. - -## Keep the logic in the write model; keep the read model thin - - The **write model** carries the full command-processing stack — input validation, business validation, invariants — and IS the `domain-driven-design` aggregate (defer there: a command loads the aggregate, @@ -315,9 +319,6 @@ that selected CQRS — not to every context in the product. - The **read model MUST return DTOs / view objects with no domain logic** — no invariants, no business rules, no validation stack. It is denormalized and query-shaped, optimized for retrieval. - -## Name commands for the business task - - Commands MUST be named for the **business task / intent** (`BookHotelRoom`, `RateProduct`, `SubmitOrder`), capturing user intent — NOT anemic field setters (`SetReservationStatusToReserved`). @@ -325,7 +326,7 @@ that selected CQRS — not to every context in the product. requests** in collaborative domains (a command scoped to a meaningful task conflicts less than a coarse "save everything"). -## Handle the consistency model explicitly +## Build - When the read model is **eventually consistent**, the UI/consumers MUST be designed for the **staleness window**: no read-your-write assumption across the @@ -339,30 +340,8 @@ that selected CQRS — not to every context in the product. - Projection / read-model update handlers MUST be **idempotent** — applying the same update event twice yields the same read-model state and fires any side effect at most once. -- A **synchronous** consistency model (read model updated in the write - transaction, same store) is the simpler choice when strong read-after-write is - required; prefer it unless the asymmetry/scale signal justifies the - eventually-consistent separation. Whichever is chosen MUST match the ADR. -## Boundary with neighbors - -- The **write model is the `domain-driven-design` aggregate** — defer there for - its shape, invariants, and the one-aggregate-per-transaction rule; this concern - asserts only that **queries bypass that model** and read from a projection, and - that CQRS applies **per bounded context**. -- When composed with **`event-sourcing`**, the **event store IS the write model / - source of truth and the projections ARE the read side** — the event - immutability, replay, snapshot, and projection-rebuild rules are - `event-sourcing`'s; do NOT restate them here. CQRS and event sourcing are - **independently adoptable**: CQRS needs no event log, event sourcing needs no - command/query split. -- The **read/write split is INTERNAL, not the wire contract** — a single - `api-style` surface (REST/GraphQL/gRPC/RPC) fronts the CQRS-structured service; - the command/query DTOs are not the exposed contract. Defer to `api-style` for - the wire interface; do not conflate command-vs-query (internal) with - mutation-vs-safe-verb (wire). - -## Quality Gates +## Test - CQRS is **scoped to the bounded context(s) that earn it** (collaborative / task-based UI / divergent read-write shape-or-scale / complex domain); @@ -387,3 +366,16 @@ that selected CQRS — not to every context in the product. eventual-consistency handling), and the **data-design** (read schema distinct from write schema + sync path) — verifiable that the selection is not silent drift. + +## Cross-cutting + +### Boundary with neighbors + +See `concern.md` for the canonical Boundary (vs `domain-driven-design`, +`event-sourcing`, `api-style`). Idempotent projection updates here are the +same property `enterprise-integration-patterns` requires of consumers under +at-least-once delivery — defer there for the cause, apply it here to the +read-model update path. When composed with event sourcing, the event store +becomes the write model / source of truth and the projections the read side; +the immutability/replay/rebuild rules are `event-sourcing`'s — not restated +here. diff --git a/docs/website/content/concerns/databricks-apps.md b/docs/website/content/concerns/databricks-apps.md index 10917a09..8ea17ebc 100644 --- a/docs/website/content/concerns/databricks-apps.md +++ b/docs/website/content/concerns/databricks-apps.md @@ -229,14 +229,11 @@ model (`unity-catalog`) — see the boundary in `concern.md`. ## Boundary with neighbors -- **vs `frontend-framework`**: the UI framework runs inside this runtime; this - concern owns hosting/identity/data wiring, not components/routing/styling. -- **vs generic `deploy-target`**: Databricks Apps **is** the deploy target here - (managed serverless) — do not stand up parallel self-hosted hosting. -- **vs `unity-catalog`**: this concern owns *that* data access flows through the - catalog and *which identity* is used; the catalog concern owns the grant model. -- **vs `security-owasp` / `auth`**: the platform supplies app service principal + - OAuth + on-behalf-of-user; compose with the app's own RBAC, do not duplicate. +See `concern.md` for the canonical Boundary (vs `frontend-framework`, generic +`deploy-target`, `unity-catalog`, `security-owasp` / `auth`). Composition in +the Databricks family: this concern hosts; `databricks-declarative-pipelines` +produces the data; `unity-catalog` governs it — each owns its piece, none +restates the others. ## Quality Gates diff --git a/docs/website/content/concerns/databricks-declarative-pipelines.md b/docs/website/content/concerns/databricks-declarative-pipelines.md index 7ed8ea6f..778a27f2 100644 --- a/docs/website/content/concerns/databricks-declarative-pipelines.md +++ b/docs/website/content/concerns/databricks-declarative-pipelines.md @@ -239,15 +239,11 @@ cross-system messaging (`enterprise-integration-patterns`) — see the boundary ## Boundary with neighbors -- **vs domain modeling**: realize the logical model here; do not re-derive it. -- **vs `enterprise-integration-patterns`**: a pipeline reading a broker source - respects EIP's at-least-once/idempotency reality at the ingestion edge, but the - expectation/DAG/incremental machinery is **not** EIP's channel/router - machinery — keep cross-system messaging in EIP. -- **vs `unity-catalog`**: this concern produces the datasets; the catalog concern - governs them (grants/ownership/lineage). -- **vs `testing`/`verification`**: expectations are **in-pipeline** data-quality - control; compose with unit/e2e testing, do not let one substitute for the other. +See `concern.md` for the canonical Boundary (vs domain modeling, +`enterprise-integration-patterns`, `unity-catalog`, `testing` / +`verification`). Composition in the Databricks family: this concern produces +the data; `unity-catalog` governs it; `databricks-apps` hosts apps that +consume it — each owns its piece, none restates the others. ## Quality Gates diff --git a/docs/website/content/concerns/demo-asciinema.md b/docs/website/content/concerns/demo-asciinema.md index 5f8674d3..4d67d99b 100644 --- a/docs/website/content/concerns/demo-asciinema.md +++ b/docs/website/content/concerns/demo-asciinema.md @@ -365,8 +365,6 @@ because they can be regenerated from scripts. Selecting this concern requires these artifacts to change (a selected concern absent from them is drift): - ADR: Asciinema scripted terminal recordings (Docker-reproducible, microsite-embedded) as the demo-reel mechanism -## ADR References - ## Practices by activity Agents working in any of these activities inherit the practices below via the bead's context digest. diff --git a/docs/website/content/concerns/demo-playwright.md b/docs/website/content/concerns/demo-playwright.md index 8dff4c03..2c463ed0 100644 --- a/docs/website/content/concerns/demo-playwright.md +++ b/docs/website/content/concerns/demo-playwright.md @@ -160,8 +160,6 @@ demonstrated. Selecting this concern requires these artifacts to change (a selected concern absent from them is drift): - ADR: Playwright reel specs (video capture, seeded data, fixed viewport) as the demo-reel mechanism -## ADR References - ## Practices by activity Agents working in any of these activities inherit the practices below via the bead's context digest. diff --git a/docs/website/content/concerns/design-patterns-gof.md b/docs/website/content/concerns/design-patterns-gof.md index 89ce5bae..aca29d78 100644 --- a/docs/website/content/concerns/design-patterns-gof.md +++ b/docs/website/content/concerns/design-patterns-gof.md @@ -172,8 +172,6 @@ Selecting this concern requires these artifacts to change (a selected concern ab - ADR: each pattern introduced against a named recurring problem + recorded intent - TD: the pattern and the variability/duplication it absorbs at the collaboration point -## ADR References - ## Practices by activity Agents working in any of these activities inherit the practices below via the bead's context digest. diff --git a/docs/website/content/concerns/domain-driven-design.md b/docs/website/content/concerns/domain-driven-design.md index 378f56c9..74821b87 100644 --- a/docs/website/content/concerns/domain-driven-design.md +++ b/docs/website/content/concerns/domain-driven-design.md @@ -200,20 +200,25 @@ object/integration mechanics that may implement a role (`design-patterns-gof`, `enterprise-integration-patterns`). When a rule below references layering, defer to onion for the *how*; this concern asserts the *what*. -## Establish the language and boundaries first +## Discover + +- Classify the area's **subdomain** as core / supporting / generic. Concentrate + modeling effort on **core**; do not deeply model a **generic** subdomain that a + bought/adopted solution already solves. + +## Frame - Name the **bounded context(s)** the work lives in, and the **ubiquitous language** of each: the precise domain terms for the concepts, used identically in specs, type names, methods, and events. A term MUST NOT mean two different things inside one context. -- Classify the area's **subdomain** as core / supporting / generic. Concentrate - modeling effort on **core**; do not deeply model a **generic** subdomain that a - bought/adopted solution already solves. - When this context consumes a foreign or legacy model, translate it through an **anti-corruption layer** — the foreign concepts MUST NOT leak into this context's model. +- Each Vernon rule is a **default**; a deliberate deviation MUST be recorded in + an ADR with its justification. -## Model the building blocks with behavior +## Design - Model concepts with **identity and lifecycle** as **entities** (equality by identity); model descriptive, replaceable concepts (money, quantity, date @@ -226,9 +231,6 @@ to onion for the *how*; this concern asserts the *what*. - Use a **factory** when constructing a complex aggregate so it is born valid; use a **domain service** (stateless) only for logic that genuinely has no home on a single entity or value object (e.g. an operation spanning two aggregates). - -## Design aggregates by their invariants - - Draw the **aggregate boundary** around exactly the true invariant — the rule that MUST stay transactionally consistent — and **no more**. The aggregate boundary IS the transactional consistency boundary. @@ -243,10 +245,8 @@ to onion for the *how*; this concern asserts the *what*. must affect another, publish a **domain event** and reconcile with **eventual consistency** outside the transaction — do not enlarge the transaction to span both roots. -- Each Vernon rule is a **default**; a deliberate deviation MUST be recorded in - an ADR with its justification. -## Keep persistence out of the domain +## Build - **Repositories return and accept aggregates** — one repository per aggregate root. ORM entities, row structs, query builders, ResultSets, and raw column @@ -255,26 +255,13 @@ to onion for the *how*; this concern asserts the *what*. - The model is expressed in the **ubiquitous language**, not in storage terms; persistence conforms to the model, not the reverse. (The dependency-inversion mechanism that makes this hold is `onion-architecture`'s practice.) - -## Enforce invariants in the domain layer - - Every true invariant is checked **inside the aggregate**, in the domain layer — **not** in a controller, request handler, route, or application service, and **not** delegated to a database constraint as the sole guard. - An application service / handler coordinates (load aggregate → call a root method → save via repository); it MUST NOT contain the business rule itself. -## Boundary with neighbors - -- For **layering** (where domain/application/infrastructure live, the dependency - rule, who defines the repository interface) defer to `onion-architecture`; do - not restate it here. -- A DDD **Factory** / **Domain Event** is a domain role; the GoF mechanic or the - messaging transport that implements it belongs to `design-patterns-gof` / - `enterprise-integration-patterns`. Name the domain role; do not specify the - mechanic here. - -## Quality Gates +## Test - Each work item's bounded context and ubiquitous language are named; domain type/method/event names match that language (no synonym drift, no term with two @@ -294,3 +281,15 @@ to onion for the *how*; this concern asserts the *what*. - Descriptive concepts with invariants (money, ranges, identifiers) are modeled as immutable value objects, not bare primitives with rules scattered at call sites. + +## Cross-cutting + +### Boundary with neighbors + +- For **layering** (where domain/application/infrastructure live, the dependency + rule, who defines the repository interface) defer to `onion-architecture`; do + not restate it here. +- A DDD **Factory** / **Domain Event** is a domain role; the GoF mechanic or the + messaging transport that implements it belongs to `design-patterns-gof` / + `enterprise-integration-patterns`. Name the domain role; do not specify the + mechanic here. diff --git a/docs/website/content/concerns/e2e-kind.md b/docs/website/content/concerns/e2e-kind.md index df962eb8..2242dd09 100644 --- a/docs/website/content/concerns/e2e-kind.md +++ b/docs/website/content/concerns/e2e-kind.md @@ -223,8 +223,6 @@ against the real service stack. Particularly valuable when: Selecting this concern requires these artifacts to change (a selected concern absent from them is drift): - TEST_PLAN: E2E against a real ephemeral kind cluster — all dependencies deployed, deterministic seed, self-contained lifecycle -## ADR References - ## Practices by activity Agents working in any of these activities inherit the practices below via the bead's context digest. diff --git a/docs/website/content/concerns/e2e-playwright.md b/docs/website/content/concerns/e2e-playwright.md index aa570251..175aeabd 100644 --- a/docs/website/content/concerns/e2e-playwright.md +++ b/docs/website/content/concerns/e2e-playwright.md @@ -25,9 +25,6 @@ e2e-framework - **Browser engines**: Chromium (default), optionally Firefox and WebKit - **Screenshot snapshots**: Visual regression detection via `toHaveScreenshot()` - **Video recording**: Full test execution videos for human review -- **Demo reel**: A dedicated slow-paced Playwright script that produces a - polished walkthrough video of the application for stakeholder review, - README embedding, or onboarding - **Trace files**: Playwright traces for debugging failures - **Test data**: Fake/seed data that exercises every meaningful UI state @@ -144,43 +141,12 @@ Playwright generates: Both directories should be gitignored. In CI, upload as artifacts. -## Demo Reel +## Demo reels -Every project with a web UI should include a Playwright demo reel script that -produces a polished walkthrough video. This replaces manual screen recordings -with a reproducible, scriptable demo. - -### Demo reel structure - -The demo reel is a single Playwright test file (e.g., `e2e/demo.spec.ts`) that: - -1. Seeds the application with comprehensive, realistic data -2. Navigates every major page and workflow at a human-readable pace -3. Uses `page.waitForTimeout()` between actions for viewing comfort -4. Produces a `.webm` video in `test-results/` that can be converted to - `.mp4` or `.gif` for embedding - -### Demo reel conventions - -- File: `e2e/demo.spec.ts` (separate from test specs) -- Viewport: 1280x720 (HD, suitable for embedding) -- Pacing: 1-2 second pauses between navigations, 2-3 seconds on key screens -- Use `test.step()` for structured logging of what each section shows -- Narrative structure: Overview (dashboard) -> Primary workflows -> Detail views -- Run with: `npx playwright test e2e/demo.spec.ts` -- Output: `test-results/demo-*/video.webm` -- The demo must run against seeded data — never against an empty state -- Re-record the demo after major UI changes - -### Converting output - -```bash -# WebM to MP4 (for README/docs) -ffmpeg -i test-results/demo-*/video.webm -c:v libx264 -preset slow -crf 22 demo.mp4 - -# WebM to GIF (for README, keep under 10MB) -ffmpeg -i test-results/demo-*/video.webm -vf "fps=10,scale=960:-1" -loop 0 demo.gif -``` +Scripted walkthrough videos are owned by **`demo-playwright`** (viewport, +pacing, narrative structure, post-processing). When a project needs a reel, +select `demo-playwright` alongside this concern; do not extend the test suite +here to double as demo capture. ## When to use @@ -196,8 +162,6 @@ Selecting this concern requires these artifacts to change (a selected concern ab - ADR: Playwright as the e2e-framework slot (real browsers, screenshots, video, traces) - TEST_PLAN: per-page + per-workflow E2E in real browsers, seeded data across UI states, committed screenshot baselines -## ADR References - ## Practices by activity Agents working in any of these activities inherit the practices below via the bead's context digest. @@ -304,18 +268,11 @@ assertion until the cue exists. - Debugging flaky tests (video shows timing/race conditions) - In CI, upload `test-results/` as a build artifact for post-hoc review -## Demo Reel -- Create `e2e/demo.spec.ts` — a single test that walks through the entire app -- Seed with realistic data that makes the demo compelling (populated states, - not empty states) -- Set viewport to 1280x720 for clean video output -- Pace with `page.waitForTimeout()`: 1-2s between clicks, 2-3s on key screens -- Use `test.step()` to narrate each section in the test output -- Structure: Dashboard overview -> key workflows -> detail pages -> settings -- Run separately: `npx playwright test e2e/demo.spec.ts` -- Output video lives in `test-results/` — convert to `.mp4` for embedding -- Re-record after significant UI changes -- The demo is documentation — keep it passing, keep it current +## Demo reels — delegated to `demo-playwright` + +Scripted walkthrough videos (viewport, pacing, narrative structure, +post-processing) belong to `demo-playwright`. Select that concern alongside +this one when a reel is needed; do not author demo specs here. ## Quality Gates - At least one core user-flow has a browser e2e test that **runs green against @@ -329,7 +286,6 @@ assertion until the cue exists. a substitute; no screenshot assertions for this cue) - Every user-facing workflow has at least one end-to-end test - Video recording is enabled (not disabled for speed) -- Demo reel script exists and produces a watchable video - No tests skip or are marked `.only` ## CI Integration diff --git a/docs/website/content/concerns/enterprise-application-patterns.md b/docs/website/content/concerns/enterprise-application-patterns.md index 3b389f4e..c7f3e2f6 100644 --- a/docs/website/content/concerns/enterprise-application-patterns.md +++ b/docs/website/content/concerns/enterprise-application-patterns.md @@ -252,13 +252,13 @@ PoEAA choices and the machinery they pull in. They sit beside the **domain-logic and data-source patterns matched to the actual complexity, recorded, and not over-built** (KISS/YAGNI). -## Choosing the domain-logic organization +## Discover - The domain-logic pattern MUST be chosen against the **assessed complexity of - the business logic**, and the choice recorded in an ADR: **Transaction Script** - for simple, mostly independent operations; **Domain Model** when the rules, - cases, and interactions are genuinely complex; **Table Module** only when a - record-set–centric stack/UI justifies it. + the business logic**: **Transaction Script** for simple, mostly independent + operations; **Domain Model** when the rules, cases, and interactions are + genuinely complex; **Table Module** only when a record-set–centric stack/UI + justifies it. - You SHOULD start with the **simplest** organization the logic warrants and refactor *toward* a Domain Model when complexity actually arrives — not stand up a Domain Model speculatively for a thin surface. @@ -266,9 +266,13 @@ recorded, and not over-built** (KISS/YAGNI). API, batch, integration) need the same operations or a single transaction/orchestration seam is required — not as a reflexive passthrough over a single caller. +- The **session-state** placement (client / server / database) SHOULD be a + recorded choice driven by state size, statelessness/affinity needs, and + survive-restart requirements — not an accident of the framework. -## Choosing the data-source pattern (Active Record vs Data Mapper) +## Frame +- The chosen domain-logic organization MUST be recorded in an ADR. - The **Active Record vs Data Mapper** decision MUST be made explicitly and **recorded in an ADR** — it MUST NOT be left as an implicit ORM default. - Choose **Active Record** when domain logic is **simple and maps closely to the @@ -279,10 +283,19 @@ recorded, and not over-built** (KISS/YAGNI). data-source pattern** — it is the mechanism that keeps the domain model ignorant of the database and lets repositories return aggregates in domain terms. Choosing Active Record under DDD MUST be justified in the ADR. + +## Design + - A rich Domain Model MUST NOT be placed on Active Record such that business rules bend to the table shape or SQL/row concerns leak into domain objects. +- The **technical-design** reflects the chosen data-source layer + domain-logic + organization, and the **data-design** reflects how that pattern maps objects to + the schema. +- **Remote Facade** and **Data Transfer Object** MUST be introduced only at a real + **process/network boundary**; they MUST NOT be added inside a single process + where fine-grained calls are free. -## Supplying the Data Mapper machinery +## Build - When a Data Mapper backs a non-trivial Domain Model, **Unit of Work** (one coordinated commit + concurrency resolution) and **Identity Map** (one in-memory @@ -291,27 +304,7 @@ recorded, and not over-built** (KISS/YAGNI). - **Lazy Load** SHOULD be used where eager-loading a whole graph is wasteful, with attention to N+1 read patterns. -## Applying distribution and session patterns only where they pay - -- **Remote Facade** and **Data Transfer Object** MUST be introduced only at a real - **process/network boundary**; they MUST NOT be added inside a single process - where fine-grained calls are free. -- The **session-state** placement (client / server / database) SHOULD be a - recorded choice driven by state size, statelessness/affinity needs, and - survive-restart requirements — not an accident of the framework. - -## Staying in your lane - -- When a construct carries **business meaning** (an aggregate, an invariant, a - ubiquitous-language name), the domain semantics belong to - `domain-driven-design`; PoEAA describes only the persistence/orchestration - **mechanics**. For **Repository** and **Service Layer**, name the domain meaning - per DDD and let this concern own the persistence/orchestration machinery. -- The patterns here fill layers; **which layer code lives in and the dependency - direction** belong to `onion-architecture` and MUST NOT be restated. A **generic - intra-process OO collaboration** is `design-patterns-gof`, not a PoEAA pattern. - -## Quality Gates +## Test - The **domain-logic organization** (Transaction Script vs Domain Model vs Table Module) is recorded in an ADR and matches the assessed complexity — no Domain @@ -334,3 +327,16 @@ recorded, and not over-built** (KISS/YAGNI). - Every pattern present is named to match the mechanic actually implemented (no one-method "Service Layer", no row-returning "Repository") and routes domain meaning, macro layering, and generic OO mechanics to their owning concerns. + +## Cross-cutting + +### Staying in your lane + +- When a construct carries **business meaning** (an aggregate, an invariant, a + ubiquitous-language name), the domain semantics belong to + `domain-driven-design`; PoEAA describes only the persistence/orchestration + **mechanics**. For **Repository** and **Service Layer**, name the domain meaning + per DDD and let this concern own the persistence/orchestration machinery. +- The patterns here fill layers; **which layer code lives in and the dependency + direction** belong to `onion-architecture` and MUST NOT be restated. A **generic + intra-process OO collaboration** is `design-patterns-gof`, not a PoEAA pattern. diff --git a/docs/website/content/concerns/enterprise-integration-patterns.md b/docs/website/content/concerns/enterprise-integration-patterns.md index 8a95f3cb..657ec83d 100644 --- a/docs/website/content/concerns/enterprise-integration-patterns.md +++ b/docs/website/content/concerns/enterprise-integration-patterns.md @@ -351,19 +351,13 @@ they reference those concerns at the seam and stay on the wire. ## Boundary with neighbors -- **vs `domain-driven-design`**: DDD decides the **fact and its invariant** — a - domain event reconciling aggregates inside a bounded context. EIP decides the - **channel, delivery guarantee, and wire shape** that carries that fact across - the boundary. Do not re-decide the domain model here; do not let DDD own the - transport. -- **vs `onion-architecture`**: messaging clients, gateways, and consumers are - **outer-ring** adapters; the inner layer declares the port, the EIP adapter - implements it. Do not restate the dependency rule; do honor it (no broker - import inward). -- **vs `design-patterns-gof`**: GoF Observer / an in-process event bus is - **not** a Message Channel substitute for a real system boundary — it has no - durability, delivery guarantee, retry, or dead-letter path. Use a channel when - the boundary is real. +See `concern.md` for the canonical Boundary (vs `domain-driven-design`, +`onion-architecture`, `design-patterns-gof`). EIP owns the at-least-once +channel-delivery model that *causes* the redelivery downstream consumers must +absorb — `concurrency-model` (in-process workers), `event-sourcing` (event +handlers), and `cqrs` (projection updaters) each apply the resulting +idempotency requirement at their own surface; this concern does not restate +their application of it. ## Quality Gates diff --git a/docs/website/content/concerns/event-sourcing.md b/docs/website/content/concerns/event-sourcing.md index 83a671ce..81fc10c9 100644 --- a/docs/website/content/concerns/event-sourcing.md +++ b/docs/website/content/concerns/event-sourcing.md @@ -322,20 +322,14 @@ not to every entity in the product. ## Boundary with neighbors -- The **aggregate, its invariants, and its emitted domain events** are - `domain-driven-design`'s — defer there for the model; this concern persists - and rehydrates that aggregate. -- The **event store is an outer-ring adapter** behind an inner-declared port — - defer to `onion-architecture` for the layering/dependency direction; do not - restate the dependency rule. -- A **stored event is not an integration message**. Publishing events to other - systems/contexts (channels, transformation, delivery) is - `enterprise-integration-patterns`' — an event handler may emit an integration - event, but do not conflate the stream-of-record with the wire message. -- The **read side / projections** are commonly the **CQRS** read model; name - CQRS as the companion, do not define the full command/query segregation here. -- Use a **real event store** (per-entity stream reads + optimistic concurrency), - not a message broker, as the system of record. +See `concern.md` for the canonical Boundary (vs `domain-driven-design`, +`enterprise-integration-patterns`, `onion-architecture`, `sample-data`, and +the **CQRS** companion). Idempotent application of events here is the same +property `enterprise-integration-patterns` requires of consumers under +at-least-once delivery — defer there for the channel-delivery rule, apply it +here to projection/read-model updates. The event store is a real event store +(per-entity stream reads + optimistic concurrency), not a message broker +standing in as the system of record. ## Quality Gates diff --git a/docs/website/content/concerns/go-std.md b/docs/website/content/concerns/go-std.md index 15bc3f2c..053bc90a 100644 --- a/docs/website/content/concerns/go-std.md +++ b/docs/website/content/concerns/go-std.md @@ -69,8 +69,6 @@ Selecting this concern requires these artifacts to change (a selected concern ab - ADR: Go + standard toolchain (gofmt, golangci-lint, gosec, govulncheck) as the language-runtime - TD: error-wrapping, context-passing, interface-in-consumer conventions; lint baseline -## ADR References - ## Practices by activity Agents working in any of these activities inherit the practices below via the bead's context digest. diff --git a/docs/website/content/concerns/hexagonal-architecture.md b/docs/website/content/concerns/hexagonal-architecture.md index ca29924e..0f55a944 100644 --- a/docs/website/content/concerns/hexagonal-architecture.md +++ b/docs/website/content/concerns/hexagonal-architecture.md @@ -193,7 +193,18 @@ The differentiator versus the sibling dependency-inversion styles (`onion` / ports** and the **plurality of adapters per port**, not on an internal concentric ring layout. -## Ports owned by the core; dependencies point inward +## Discover + +- Apply full ports-and-adapters only when the **selection signals** in + `concern.md` hold — **multiple symmetric driving and/or driven adapters** + around one core. For a thin CRUD app with one UI and one datastore and no + prospect of a second adapter, the port ceremony is over-engineering — prefer + `classic-layered`, recorded as the `architecture-style` choice. +- Per KISS/YAGNI, do NOT introduce a port for a boundary that has exactly one + adapter and no realistic prospect of a second, **unless** the port is needed + to keep the core driveable/testable in isolation. + +## Design - Code MUST be organized into an **application core** plus **adapters**, with a discoverable mapping from core / driving-adapter / driven-adapter to @@ -205,9 +216,6 @@ concentric ring layout. - A **driven (secondary) port** — the interface for persistence or an external system — MUST be declared **in the core** and implemented in a secondary adapter. The port interface MUST NOT live in the adapter's package. - -## Symmetry of driving and driven sides - - **Driving (primary) adapters** (HTTP, CLI, queue consumer, GUI, test harness) MUST call the core only through **driving ports**. **Driven (secondary) adapters** (SQL/ORM, message bus, email, third-party clients) MUST be invoked @@ -219,8 +227,12 @@ concentric ring layout. attach a second adapter (a test harness on a driving port; an in-memory fake on a driven port) without modifying the core. A port whose single implementation is hard-wired is a defect. +- Data crossing a port MUST be expressed in the **core's own terms** (core + objects / DTOs the core owns). Transport shapes (HTTP request/response + objects, ORM rows) MUST be translated **inside the adapter** and MUST NOT + cross the port into the core. -## Configurable dependency at the boundary +## Build - Concrete adapters MUST be instantiated and wired to the core's ports **at configuration time by the composition root** — the only place that names @@ -228,40 +240,12 @@ concentric ring layout. adapter. - Swapping an adapter (different datastore, CLI instead of HTTP) MUST require changes only in that adapter and the wiring, never in the core. -- Data crossing a port MUST be expressed in the **core's own terms** (core - objects / DTOs the core owns). Transport shapes (HTTP request/response - objects, ORM rows) MUST be translated **inside the adapter** and MUST NOT - cross the port into the core. - -## Keep the core driveable in isolation - - The core SHOULD be **exercisable through ports with no real devices or databases present** — driven by a test (primary) adapter and backed by fake (secondary) adapters. (Writing those tests is the `testing` concern; this practice only requires the port seams to exist on both sides.) -## Match the discipline to the product (avoid over-engineering) - -- Apply full ports-and-adapters only when the **selection signals** in - `concern.md` hold — **multiple symmetric driving and/or driven adapters** - around one core. For a thin CRUD app with one UI and one datastore and no - prospect of a second adapter, the port ceremony is over-engineering — prefer - `classic-layered`, recorded as the `architecture-style` choice. -- Per KISS/YAGNI, do NOT introduce a port for a boundary that has exactly one - adapter and no realistic prospect of a second, **unless** the port is needed - to keep the core driveable/testable in isolation. - -## Boundary with sibling concerns - -- The **contents** of the core are governed by `domain-driven-design`, not here. - Verify the model sits inside the hexagon behind ports; do not restate DDD - modeling rules. -- Object-level collaboration patterns inside a layer are `design-patterns-gof`; - between-system integration is `enterprise-integration-patterns` (its messaging - endpoints are driven adapters under this concern). Hexagonal governs only the - macro port boundary across the codebase. - -## Quality Gates +## Test - Import-graph check: the application core has **zero** dependency edges to any adapter, framework, transport, or driver package. @@ -280,3 +264,15 @@ concentric ring layout. - Selection-fit check: multiple symmetric driving/driven adapters justify the port ceremony; ports-and-adapters is not wrapped around a single-UI, single-datastore thin-CRUD app (else re-select `classic-layered`). + +## Cross-cutting + +### Boundary with sibling concerns + +- The **contents** of the core are governed by `domain-driven-design`, not here. + Verify the model sits inside the hexagon behind ports; do not restate DDD + modeling rules. +- Object-level collaboration patterns inside a layer are `design-patterns-gof`; + between-system integration is `enterprise-integration-patterns` (its messaging + endpoints are driven adapters under this concern). Hexagonal governs only the + macro port boundary across the codebase. diff --git a/docs/website/content/concerns/hugo-hextra.md b/docs/website/content/concerns/hugo-hextra.md index b332fe11..7ce97dc0 100644 --- a/docs/website/content/concerns/hugo-hextra.md +++ b/docs/website/content/concerns/hugo-hextra.md @@ -241,8 +241,6 @@ Selecting this concern requires these artifacts to change (a selected concern ab - ADR: Hugo (extended) + Hextra theme via Hugo Modules, GitHub Pages deploy, as the microsite stack - TD: website/ layout, pinned theme/Hugo versions, Hugo config (enableGitInfo, unsafe HTML) -## ADR References - ## Practices by activity Agents working in any of these activities inherit the practices below via the bead's context digest. diff --git a/docs/website/content/concerns/i18n-icu.md b/docs/website/content/concerns/i18n-icu.md index d15e1dc4..2c56e664 100644 --- a/docs/website/content/concerns/i18n-icu.md +++ b/docs/website/content/concerns/i18n-icu.md @@ -41,8 +41,6 @@ Selecting this concern requires these artifacts to change (a selected concern ab - TD: ICU MessageFormat, externalized string catalogs, locale-aware formatting, declared default locale - DESIGN_SYSTEM: bidirectional (LTR/RTL) text support -## ADR References - ## Practices by activity Agents working in any of these activities inherit the practices below via the bead's context digest. diff --git a/docs/website/content/concerns/k8s-kind.md b/docs/website/content/concerns/k8s-kind.md index 5026a5aa..ed5ab796 100644 --- a/docs/website/content/concerns/k8s-kind.md +++ b/docs/website/content/concerns/k8s-kind.md @@ -16,6 +16,16 @@ infrastructure ## Areas infra +## Boundary + +This concern is a **runtime implementation** that hosts whatever deployables +`deployment-topology` chose to ship (one modular monolith or many +microservices — both run here). It owns cluster / Helm chart / image-build / +local-kind workflow mechanics. It does **not** decide the deployable count +or seams (`deployment-topology`), the per-process operational contract each +deployable honors (`twelve-factor`), or the telemetry the deployables emit +(`o11y-otel`). + ## Components - **Local cluster**: `kind` (Kubernetes in Docker) — NOT docker-compose @@ -45,8 +55,6 @@ Selecting this concern requires these artifacts to change (a selected concern ab - ADR: Kubernetes + kind (Helm, reproducible images) as deployment topology — not docker-compose - IMPLEMENTATION_PLAN: Helm charts + env values files, image-build/tag, kind create + install + port-forward workflow -## ADR References - ## Practices by activity Agents working in any of these activities inherit the practices below via the bead's context digest. diff --git a/docs/website/content/concerns/mcp-server.md b/docs/website/content/concerns/mcp-server.md index e52b2726..cee88003 100644 --- a/docs/website/content/concerns/mcp-server.md +++ b/docs/website/content/concerns/mcp-server.md @@ -281,11 +281,14 @@ styles), re-specify the general threat model (`security-owasp`), or govern asynchronous messaging (`enterprise-integration-patterns`); they reference those at the seam and stay on the MCP surface. -## Decide MCP is the right surface, then model the primitives +## Discover - Confirm the **consumer is an LLM agent / AI client** consuming tools, data, or prompts for autonomous use (the `api-style` selection signal for MCP). If the consumer is a human or another service, the surface is REST/gRPC/etc., not MCP. + +## Frame + - Map each capability to the **right primitive by control surface**: - **Tool** — a **model-invoked action with effects** (write, send, call an external API); one operation, **typed JSON-Schema input + output**. @@ -297,7 +300,7 @@ at the seam and stay on the MCP surface. passive data through a tool the model must decide to call. Record the exposed primitives in an ADR. -## Write each tool description as the model's contract +## Design - Give every tool a **precise, honest, scoped description** stating what it does, when to use it, and what it does NOT do — the model selects tools on this text @@ -308,9 +311,6 @@ at the seam and stay on the MCP surface. - Do **not** embed steering instructions in a description beyond describing the tool, and treat **consumed** third-party tool descriptions as untrusted (tool poisoning, below). - -## Gate destructive tools behind human-in-the-loop - - A tool with **irreversible or high-impact effect** (delete, send, pay, deploy, mutate external state) **requires explicit user confirmation at invocation** — a consent the user sees *before* the effect happens. Read-only/safe tools may @@ -318,8 +318,12 @@ at the seam and stay on the MCP surface. decision. - Surface tool execution (approval dialog and/or activity log) so the user can **see what a tool does before authorizing it**. +- Use **stdio** for a **local** server (runs as a subprocess, JSON-RPC over + stdin/stdout, credentials from the environment) — preferred where the client + runs the server locally. Use **Streamable HTTP** for a **remote / multi-client** + server (single MCP endpoint, POST/GET, optional SSE streaming). -## Validate every tool input; scope every tool least-privilege +## Build - **Validate and coerce each tool's arguments against its JSON Schema at the boundary** before acting — the caller is an LLM that can emit malformed, @@ -329,9 +333,6 @@ at the seam and stay on the MCP surface. `full-access` / omnibus scopes, no bundling unrelated privileges up front. Prefer **progressive scope elevation** (start minimal, elevate on first privileged use) so a stolen token's blast radius stays small. - -## Defend against tool poisoning and rug-pulls (when consuming/aggregating servers) - - Treat the **descriptions and schemas of tools this product consumes** as a prompt-injection / supply-chain surface: unless the source server is trusted, do not let consumed tool metadata steer the agent. Pin/verify metadata from @@ -339,20 +340,6 @@ at the seam and stay on the MCP surface. - Detect **changed tool definitions** and **re-consent** rather than silently honoring a tool whose behavior/description changed after the user approved an earlier form (rug-pull). - -## Choose the transport and lock it down - -- Use **stdio** for a **local** server (runs as a subprocess, JSON-RPC over - stdin/stdout, credentials from the environment) — preferred where the client - runs the server locally. Use **Streamable HTTP** for a **remote / multi-client** - server (single MCP endpoint, POST/GET, optional SSE streaming). -- For a local HTTP server, **bind to localhost** (not `0.0.0.0`) and **validate - the `Origin` header** to prevent DNS-rebinding. Restrict a local HTTP server's - access (auth token, or IPC/unix socket) so other local processes cannot drive - it. - -## Authorize the HTTP transport with OAuth 2.1 — validate audience, never pass through - - An **HTTP-transport** MCP server is an **OAuth 2.1 resource server**: publish protected-resource metadata (RFC 9728), require `Authorization: Bearer` on every request, and **validate that each token was issued specifically for this @@ -370,25 +357,14 @@ at the seam and stay on the MCP surface. random, **user-bound** session IDs (`:`) — never a session ID as proof of identity, never predictable session IDs. -## Boundary with neighbors +## Deploy -- **vs `api-style`**: api-style selects the interface style and *places MCP* - among them (the agent-facing option) and records the selection; this concern - holds the MCP-specific discipline once MCP is chosen. Do not re-run the - style-selection guide here. -- **vs `security-owasp`**: security-owasp owns the baseline threat model (OWASP - Top 10, injection, access control, secrets, TLS); this concern adds the - **agent-exposure** layer (tool poisoning, rug-pull, confused-deputy, token - passthrough, over-broad tool scopes). Compose; do not restate the Top 10. -- **vs `enterprise-integration-patterns`**: MCP is a **synchronous** tool/ - resource interface the agent calls and awaits; a tool that *enqueues* work - writes to an EIP channel — the `tools/call` is this concern, the channel is - EIP's. Do not model MCP as an async message bus. -- **vs `auth`/`auth-local-sessions`**: those own the product's own user login; - this concern owns the **OAuth 2.1 agent-token boundary** for the MCP HTTP - transport (audience binding, no passthrough). Compose; do not conflate. +- For a local HTTP server, **bind to localhost** (not `0.0.0.0`) and **validate + the `Origin` header** to prevent DNS-rebinding. Restrict a local HTTP server's + access (auth token, or IPC/unix socket) so other local processes cannot drive + it. -## Quality Gates +## Test - **Each capability is modeled to the right primitive** — model-invoked actions with effects are **tools** (typed JSON-Schema input/output, single-purpose), @@ -414,3 +390,23 @@ at the seam and stay on the MCP surface. - **Consumed/aggregated third-party tool metadata is treated as untrusted** — defense against tool poisoning, and re-consent on changed tool definitions (rug-pull) where the product proxies other MCP servers. + +## Cross-cutting + +### Boundary with neighbors + +- **vs `api-style`**: api-style selects the interface style and *places MCP* + among them (the agent-facing option) and records the selection; this concern + holds the MCP-specific discipline once MCP is chosen. Do not re-run the + style-selection guide here. +- **vs `security-owasp`**: security-owasp owns the baseline threat model (OWASP + Top 10, injection, access control, secrets, TLS); this concern adds the + **agent-exposure** layer (tool poisoning, rug-pull, confused-deputy, token + passthrough, over-broad tool scopes). Compose; do not restate the Top 10. +- **vs `enterprise-integration-patterns`**: MCP is a **synchronous** tool/ + resource interface the agent calls and awaits; a tool that *enqueues* work + writes to an EIP channel — the `tools/call` is this concern, the channel is + EIP's. Do not model MCP as an async message bus. +- **vs `auth`/`auth-local-sessions`**: those own the product's own user login; + this concern owns the **OAuth 2.1 agent-token boundary** for the MCP HTTP + transport (audience binding, no passthrough). Compose; do not conflate. diff --git a/docs/website/content/concerns/multi-tenancy.md b/docs/website/content/concerns/multi-tenancy.md index bc160479..c8e01659 100644 --- a/docs/website/content/concerns/multi-tenancy.md +++ b/docs/website/content/concerns/multi-tenancy.md @@ -31,38 +31,19 @@ data, api This concern owns **isolation between tenants** — that tenant A can never read or write tenant B's data, on any path, and the architecture that delivers that guarantee (which resources are shared vs dedicated). It is composable and does -**not** fill a slot. Three neighbours stay distinct: - -- **`auth`** owns **who you are** — signup, login, sessions, and that a real - principal exists and is scoped to its account. `auth` establishes the - authenticated principal and the account/tenant it belongs to; **multi-tenancy - consumes that principal** and enforces that *every data access stays inside - that tenant's boundary*. Auth answers "are you signed in as someone in - tenant T?"; multi-tenancy answers "given that, can any code path reach data - outside T?". `auth` already states isolation-through-the-principal as a - requirement; this concern owns the **enforcement and isolation model** behind - it and makes the cross-tenant guard reviewer-checkable. Do not restate - signup/session lifecycle here. -- **`authorization-model`** (a sibling concern) owns **what a user may do - *within* a tenant** — roles, permissions, and the actions a principal is - allowed to perform on resources they can already legitimately reach. - Authorization is *intra-tenant* ("may this member delete this project in - *their own* tenant?"). Multi-tenancy is *inter-tenant* ("is this project even - in the caller's tenant at all?"). A correct authorization check on a record - that belongs to the wrong tenant is still a cross-tenant leak — the tenant - predicate must hold *before* the permission check is even meaningful. Compose - the two; do not fold authorization rules into this concern. -- **`security-owasp`** owns general hardening — injection, CSRF, secret - handling, input validation, TLS. Cross-tenant access is an instance of OWASP - **Broken Access Control / IDOR**; this concern is the tenant-specific, - reviewer-checkable refinement of that hardening. `security-owasp` says - "authorization checked on every protected endpoint"; multi-tenancy says - "every data access carries the tenant predicate, derived from the principal". - -This concern owns the one thing those do not state: **there is no query path -that resolves a record without the tenant predicate, and tenant identity is -derived from the authenticated principal — never from a client-supplied id -alone.** +**not** fill a slot. + +For the family ownership table (auth / authorization-model / multi-tenancy / +security-owasp, plus the admin-console and unity-catalog neighbors), and the +ordering invariants (tenant predicate precedes permission; authn precedes +authz), see [README-auth-family.md](../readme-auth-family/). + +This concern owns the one thing the rest of the family does not state: **there +is no query path that resolves a record without the tenant predicate, and +tenant identity is derived from the authenticated principal — never from a +client-supplied id alone.** Cross-tenant access is the OWASP Broken Access +Control / IDOR case `security-owasp` names at the umbrella level; this concern +is the tenant-specific, reviewer-checkable refinement. ## Components @@ -213,12 +194,12 @@ mapping) and the tenant lifecycle (provisioning + offboarding). Agents working in any of these activities inherit the practices below via the bead's context digest. These practices govern **isolation between tenants** — that no code path lets -one tenant reach another's data. They sit alongside `auth` (who you are), -`authorization-model` (what you may do *within* a tenant), and `security-owasp` -(general hardening; cross-tenant access is Broken Access Control / IDOR). Their -one job is to make the **cross-tenant leak** unreachable and **reviewer-checkable**. -Each MUST/SHOULD below is written so a reviewer can confirm or refute it against -the diff and the running system. +one tenant reach another's data. For the family ownership table (auth / +authorization-model / multi-tenancy / security-owasp) see +[README-auth-family.md](../readme-auth-family/). Their one job is to make the +**cross-tenant leak** unreachable and **reviewer-checkable**. Each MUST/SHOULD +below is written so a reviewer can confirm or refute it against the diff and +the running system. ## Choose and record the isolation model (Frame / Design) @@ -304,19 +285,6 @@ the diff and the running system. - Offboarding **MUST** be able to export and delete a single tenant's data without affecting other tenants. -## Boundary with neighbors - -- For **who the principal is** (signup, login, sessions, account bootstrap) - defer to `auth`; do not restate it here. This concern consumes the principal. -- For **what a user may do within their tenant** (roles, permissions) defer to - `authorization-model`; this concern only guarantees the record is in the - caller's tenant in the first place. -- For general hardening (injection, CSRF, secrets, TLS, parameterized queries) - defer to `security-owasp`; cross-tenant access is its Broken Access Control / - IDOR case, refined here for tenants. -- For the tenant as part of the domain model (tenant as aggregate identity, - invariants scoped to a tenant) compose with `domain-driven-design`. - ## Quality Gates - Isolation model recorded in an ADR with its trade-off justification — not diff --git a/docs/website/content/concerns/o11y-otel.md b/docs/website/content/concerns/o11y-otel.md index bc9487d9..e68e4bd3 100644 --- a/docs/website/content/concerns/o11y-otel.md +++ b/docs/website/content/concerns/o11y-otel.md @@ -41,8 +41,6 @@ Selecting this concern requires these artifacts to change (a selected concern ab - ADR: OpenTelemetry (traces/metrics/logs) as the observability standard - TD: RED metrics on endpoints, trace-context propagation, structured JSON logs with correlation IDs -## ADR References - ## Practices by activity Agents working in any of these activities inherit the practices below via the bead's context digest. diff --git a/docs/website/content/concerns/onion-architecture.md b/docs/website/content/concerns/onion-architecture.md index 718a8c3c..85bd3784 100644 --- a/docs/website/content/concerns/onion-architecture.md +++ b/docs/website/content/concerns/onion-architecture.md @@ -180,7 +180,18 @@ object-level patterns (`design-patterns-gof`), not between-system messaging (`enterprise-integration-patterns`). Where DDD is also selected, its domain model is exactly what lives in the core ring described here. -## Layering and dependency direction +## Discover + +- Apply the full ring layering only when the **selection signals** in + `concern.md` hold (non-trivial domain logic, swappable infrastructure, or a + testable-domain requirement). For **thin CRUD / forms-over-data** with little + behavior, the ring ceremony is over-engineering — prefer the + `classic-layered` slot filler, recorded as the `architecture-style` choice. +- Per KISS/YAGNI, do not introduce an interface for a boundary that has exactly + one implementation and no realistic prospect of a second **unless** it is + needed to keep the domain testable in isolation. + +## Design - The code MUST be organized into concentric rings — **domain model core → domain services → application services → outer ring (infrastructure / UI / @@ -195,9 +206,6 @@ model is exactly what lives in the core ring described here. dependency still points inward** — a controller/handler depends on an application service; the application/domain layers MUST NOT depend on the controller. - -## Dependency inversion at the boundary - - Interfaces the core needs (repositories, gateways, ports, notifiers) MUST be **declared in the inner layer** (domain or application) in domain terms, and **implemented in the outer ring**. The interface and its concrete @@ -205,16 +213,16 @@ model is exactly what lives in the core ring described here. - Inner-layer code MUST depend on these interfaces, never on the concrete outer-ring class. Inner code MUST NOT `new`/construct or `import` a concrete infrastructure implementation directly. -- Concrete outer-ring implementations MUST be **injected at runtime by the - composition root** (the entrypoint / DI container / `main`). The composition - root is the **only** place that names concrete infrastructure types. - Data crossing a boundary MUST be expressed in **inner-layer terms** — domain objects or DTOs the inner layer owns. ORM rows, framework request/response objects, and other outer-ring shapes MUST NOT leak into the domain or application layers; translate at the boundary. -## Keep the core infrastructure-agnostic +## Build +- Concrete outer-ring implementations MUST be **injected at runtime by the + composition root** (the entrypoint / DI container / `main`). The composition + root is the **only** place that names concrete infrastructure types. - The domain and application layers SHOULD be **buildable/exercisable without any infrastructure present** — substituting a fake/stub adapter for each inner-layer interface should compile and run. (Writing those tests is the @@ -224,28 +232,7 @@ model is exactly what lives in the core ring described here. framework) SHOULD require changes only in the outer ring and the composition root, not in the core. -## Match the discipline to the product (avoid over-engineering) - -- Apply the full ring layering only when the **selection signals** in - `concern.md` hold (non-trivial domain logic, swappable infrastructure, or a - testable-domain requirement). For **thin CRUD / forms-over-data** with little - behavior, the ring ceremony is over-engineering — prefer the - `classic-layered` slot filler, recorded as the `architecture-style` choice. -- Per KISS/YAGNI, do not introduce an interface for a boundary that has exactly - one implementation and no realistic prospect of a second **unless** it is - needed to keep the domain testable in isolation. - -## Boundary with sibling concerns - -- The **contents** of the core (aggregates, entities, value objects, - invariants, ubiquitous language) are governed by `domain-driven-design`, not - here. Do not restate DDD modeling rules in Onion review; do verify the model - sits in the core ring with inward-only dependencies. -- Object-level collaboration patterns inside a layer are `design-patterns-gof`; - between-system integration is `enterprise-integration-patterns`. Onion only - governs the macro dependency structure across the codebase. - -## Quality Gates +## Test - Import-graph check: the domain layer has **zero** dependency edges to infrastructure / framework / ORM / web / outer-ring packages. @@ -261,3 +248,15 @@ model is exactly what lives in the core ring described here. domain or application layers — boundary translation is present. - The `architecture-style` selection fits the product: ring layering is not wrapped around a thin-CRUD app (else re-select `classic-layered`). + +## Cross-cutting + +### Boundary with sibling concerns + +- The **contents** of the core (aggregates, entities, value objects, + invariants, ubiquitous language) are governed by `domain-driven-design`, not + here. Do not restate DDD modeling rules in Onion review; do verify the model + sits in the core ring with inward-only dependencies. +- Object-level collaboration patterns inside a layer are `design-patterns-gof`; + between-system integration is `enterprise-integration-patterns`. Onion only + governs the macro dependency structure across the codebase. diff --git a/docs/website/content/concerns/python-uv.md b/docs/website/content/concerns/python-uv.md index 731ae60a..4147e37c 100644 --- a/docs/website/content/concerns/python-uv.md +++ b/docs/website/content/concerns/python-uv.md @@ -54,8 +54,6 @@ Selecting this concern requires these artifacts to change (a selected concern ab - ADR: Python 3.12+ + uv (ruff, pyright, pytest) as the language-runtime - TD: pyproject.toml layout, dependency-group conventions, branch-coverage floor -## ADR References - ## Practices by activity Agents working in any of these activities inherit the practices below via the bead's context digest. diff --git a/docs/website/content/concerns/react-nextjs.md b/docs/website/content/concerns/react-nextjs.md index 4a157aa0..c1002dac 100644 --- a/docs/website/content/concerns/react-nextjs.md +++ b/docs/website/content/concerns/react-nextjs.md @@ -23,7 +23,9 @@ frontend-framework - **UI Framework**: React 19 — functional components and hooks only - **Meta-framework**: Next.js 15 — App Router (not Pages Router) -- **Component library**: shadcn/ui (copied components, not npm dependency) + Radix UI primitives +- **UI primitives / component library**: see `ux-radix` concern (canonical + owner of the Radix + shadcn/ui prescription) — this concern does NOT + prescribe a component library directly - **Styling**: Tailwind CSS 4 — utility-first, no CSS-in-JS - **Forms**: react-hook-form + @hookform/resolvers with Zod schemas - **Data tables**: TanStack React Table 8 — headless, sortable, filterable, virtualizable @@ -38,7 +40,9 @@ frontend-framework - No `React.FC` type annotation — use plain function signatures with typed props - Forms must use react-hook-form with uncontrolled components — no `useState` per field - Validation schemas live in the shared package and are reused on frontend and backend -- shadcn/ui components are copied into the project — do not install as npm dependency +- UI primitives and component-library choice are governed by the `ux-radix` + concern (canonical owner of shadcn/ui + Radix); compose `ux-radix` when this + concern is selected for UI work - Tailwind config extends the design system tokens (colors, spacing, typography) - E2E tests use Playwright, not Cypress or Selenium @@ -49,7 +53,8 @@ frontend-framework - `React.FC` or `React.FunctionComponent` → use plain typed functions - `useState` for every form field → must use react-hook-form - `styled-components`, `emotion`, or `css-modules` → use Tailwind utility classes -- `@shadcn/ui` as npm dependency → must be copied components +- Re-prescribing shadcn/ui or Radix primitives here → defer to `ux-radix` + (canonical owner); this concern only references that prescription - `cypress` or `selenium` → use Playwright - `getServerSideProps` or `getStaticProps` → use Server Components or route handlers - Inline `fetch` in components without error/loading states → use data fetching pattern with Suspense boundaries @@ -57,21 +62,25 @@ frontend-framework ## When to use React + Next.js frontend applications. Compose with `typescript-bun` for the -base TypeScript and Bun runtime concern. This concern adds React-specific UI -patterns, component library conventions, and E2E testing requirements. +base TypeScript and Bun runtime concern, and compose with `ux-radix` for UI +primitives and component-library prescription (shadcn/ui + Radix). This +concern adds React-specific framework patterns (App Router, Server Components, +forms, data tables) and E2E testing requirements; it does NOT prescribe a +component library. ## Artifact Impact Selecting this concern requires these artifacts to change (a selected concern absent from them is drift): -- ADR: React 19 + Next.js App Router as the frontend-framework slot; shadcn/Radix/Tailwind +- ADR: React 19 + Next.js App Router as the frontend-framework slot; Tailwind for styling (component-library prescription is owned by `ux-radix`) - TD: App Router + Server Components, react-hook-form + Zod, shared validation schemas -- DESIGN_SYSTEM: Tailwind config extends design-system tokens; shadcn/ui component conventions +- DESIGN_SYSTEM: Tailwind config extends design-system tokens (component-library conventions live with `ux-radix`) - TEST_PLAN: Playwright E2E (not Cypress/Selenium) ## ADR References - ADR-010: Frontend validation architecture (Zod shared schemas) -- ADR-011: ERP component library and navigation (shadcn/ui + Radix + Tailwind) +- ADR-011: ux-radix owns the Radix and shadcn component-library prescription; + this concern references rather than re-prescribing ## Practices by activity @@ -86,8 +95,10 @@ Agents working in any of these activities inherit the practices below via the be - Use Next.js App Router: layouts in `app/layout.tsx`, pages in `app/page.tsx`, route groups with `(groupName)/` - Default to React Server Components — add `"use client"` only for interactive components (forms, modals, dropdowns, client state) - Component hierarchy: page (server) → layout (server) → interactive widget (client) -- shadcn/ui for UI primitives — copy components via `npx shadcn@latest add `, customize in `components/ui/` -- Radix UI for accessible headless primitives underlying shadcn +- UI primitives and component library: see the `ux-radix` concern — it is the + canonical owner of the shadcn/ui + Radix prescription (copy-not-install, + customization, primitive selection). Compose `ux-radix` when this concern + is selected for UI work. - Design tokens (colors, spacing, typography) in `tailwind.config.ts` — components reference tokens, not raw values - Forms: one Zod schema per entity in `@apogee/shared`, resolved via `@hookform/resolvers/zod` - Data tables: TanStack React Table with column definitions typed against shared schemas @@ -99,7 +110,8 @@ Agents working in any of these activities inherit the practices below via the be - Hooks: `use` prefix, one file per hook in `hooks/` directory - Server Components: default export async functions, fetch data directly - Client Components: `"use client"` directive at top of file, minimize scope -- Tailwind classes: use `cn()` utility (from shadcn) for conditional classes — no `classnames` or `clsx` separately +- Tailwind classes: use a `cn()` utility for conditional classes — no + `classnames` or `clsx` used separately - Forms pattern: ```tsx const form = useForm({ resolver: zodResolver(schema) }); @@ -145,6 +157,8 @@ Agents working in any of these activities inherit the practices below via the be ## Accessibility - All interactive elements must have accessible labels (aria-label, aria-labelledby, or visible text) -- shadcn/ui + Radix provide WCAG 2.1 AA keyboard and screen reader support by default — do not override with custom handlers that break accessibility +- Accessible UI-primitive behavior (keyboard, focus, screen reader) is + governed by the `ux-radix` concern — do not override Radix/shadcn handlers + in ways that break the WAI-ARIA contract - Color contrast must meet WCAG AA (4.5:1 for normal text, 3:1 for large text) - Forms must associate labels with inputs and display validation errors accessibly diff --git a/docs/website/content/concerns/resilience.md b/docs/website/content/concerns/resilience.md index 04280e7d..5c1d2715 100644 --- a/docs/website/content/concerns/resilience.md +++ b/docs/website/content/concerns/resilience.md @@ -391,24 +391,11 @@ metrics/traces these guards emit into). They reference those concerns at the sea ## Boundary with neighbors -- **vs `enterprise-integration-patterns`**: EIP owns **async-channel** - resilience — a Dead Letter Channel for poison messages, an Idempotent Receiver - because an at-least-once *channel* will redeliver, Guaranteed Delivery against a - broker crash. This concern owns **synchronous-call** resilience — the timeout, - breaker, bulkhead, and fallback on a blocking in-the-call-path dependency. - **Idempotency is shared but split:** EIP's is the receiver's defense against - channel redelivery; this concern's is the **precondition that makes a - synchronous auto-retry safe.** When the boundary is a queue/broker, use EIP; - when it is a blocking synchronous call, use resilience. Do not duplicate. -- **vs `verification`**: this concern states the behavior that must be true (the - breaker opens, the timeout fires, the fallback serves); `verification` is the - gate that refuses "done" until those guard branches were **exercised against the - running system**. Hand the open/half-open and timeout paths to the verification - evidence gate; do not restate the gate here. -- **vs `o11y-otel`**: this concern requires guard activity to be **observable**; - `o11y-otel` owns the metric/trace/log plumbing that carries it. Emit - breaker-state and timeout/shed events into that pipeline; do not re-specify the - pipeline. +See `concern.md` for the canonical Boundary (vs `enterprise-integration-patterns`, +`verification`, `o11y-otel`). These practices stay on synchronous-call +stability; defer to the neighbor named there for async-channel resilience +(EIP), the evidence gate that proves the guards ran (`verification`), and the +telemetry pipeline that carries breaker/timeout/shed signals (`o11y-otel`). ## Quality Gates diff --git a/docs/website/content/concerns/rust-cargo.md b/docs/website/content/concerns/rust-cargo.md index 8a48589d..5a111a4d 100644 --- a/docs/website/content/concerns/rust-cargo.md +++ b/docs/website/content/concerns/rust-cargo.md @@ -73,8 +73,6 @@ Selecting this concern requires these artifacts to change (a selected concern ab - ADR: Rust + Cargo workspace (clippy, fmt, cargo-deny/machete, pinned toolchain) as the language-runtime - TD: workspace lints, error-handling (thiserror/anyhow), unsafe policy, profile conventions -## ADR References - ## Practices by activity Agents working in any of these activities inherit the practices below via the bead's context digest. diff --git a/docs/website/content/concerns/sample-data.md b/docs/website/content/concerns/sample-data.md index 611d90ce..37d19748 100644 --- a/docs/website/content/concerns/sample-data.md +++ b/docs/website/content/concerns/sample-data.md @@ -130,7 +130,7 @@ with — the data that makes it feel real and drives every UI state. They do not govern test data: `testing` owns fixtures, factories, and assertion-specific data (see the boundary in `concern.md`). -## Choose the faker (follows the tech stack) +## Frame - Use the **semantic fake-data library** that is the named default for the active runtime concern: @@ -141,7 +141,7 @@ data (see the boundary in `concern.md`). - Add it as a project dependency at a **pinned version**. The library is the default generator, not the source of determinism. -## Make generation deterministic +## Design Determinism comes from how the generator is driven, not from which one: @@ -152,9 +152,6 @@ Determinism comes from how the generator is driven, not from which one: - Generate in a **deterministic order** — no reliance on map/set iteration order or wall-clock time. - Record/print the seed so any dataset can be reproduced exactly. - -## Generate varied shapes and edge cases - - Produce **varied** rows — vary string lengths, optional-field presence, value ranges, and timestamps. Never N copies of one row. - Cover the **schema/domain-relevant edge cases** so every UI state renders: @@ -167,7 +164,7 @@ Determinism comes from how the generator is driven, not from which one: - Keep edge cases **bounded** — realistic extremes the schema and domain allow, not malformed data the product need not accept. -## Treat the seed script as a governed deliverable +## Build - The seed script is **idempotent and re-runnable**: running it again converges to the same governed dataset rather than appending duplicates. @@ -181,14 +178,7 @@ Determinism comes from how the generator is driven, not from which one: inside the same governed script for boundary cases you must guarantee. Avoid thin hardcoded-only data as the *whole* dataset. -## Boundary with testing - -- `sample-data` populates the **running product**; `testing` constructs - throwaway data for a single assertion. Do not seed the app from test fixtures, - and do not assert product behavior against the demo seed instead of - purpose-built test data. - -## Quality Gates +## Test - A semantic faker library is a pinned dependency for the data-backed product. - A governed, idempotent seed script exists and populates the running product. @@ -198,3 +188,12 @@ Determinism comes from how the generator is driven, not from which one: ordered generation. - Sample data is clearly synthetic (no PII) and the seed never targets production. + +## Cross-cutting + +### Boundary with testing + +- `sample-data` populates the **running product**; `testing` constructs + throwaway data for a single assertion. Do not seed the app from test fixtures, + and do not assert product behavior against the demo seed instead of + purpose-built test data. diff --git a/docs/website/content/concerns/scala-sbt.md b/docs/website/content/concerns/scala-sbt.md index e39083d1..e397a846 100644 --- a/docs/website/content/concerns/scala-sbt.md +++ b/docs/website/content/concerns/scala-sbt.md @@ -51,8 +51,6 @@ Selecting this concern requires these artifacts to change (a selected concern ab - ADR: Scala + sbt (scalafmt, scalafix, ScalaTest, ZIO where applicable) as the language-runtime - TD: sbt-dynver versioning, format/lint gates, build-cache and concurrency conventions -## ADR References - ## Practices by activity Agents working in any of these activities inherit the practices below via the bead's context digest. diff --git a/docs/website/content/concerns/security-owasp.md b/docs/website/content/concerns/security-owasp.md index 5bc2d079..104105b3 100644 --- a/docs/website/content/concerns/security-owasp.md +++ b/docs/website/content/concerns/security-owasp.md @@ -16,6 +16,22 @@ security ## Areas all +## Boundary + +This concern owns the **hardening posture** for the product (OWASP Top 10: +injection, CSRF, secret handling, input validation, TLS, dependency auditing, +parameterized queries, error-detail leakage) and the **audit-logging policy** +for security-relevant events — what to log on authz denial, login failure, +privilege escalation. It is composable, applies across every area, and does +not fill a slot. + +For the family ownership table (auth / authorization-model / multi-tenancy / +security-owasp, plus the admin-console and unity-catalog neighbors) see +[README-auth-family.md](../readme-auth-family/). Broken Access Control is the +OWASP umbrella; `authorization-model` is the per-handler model that prevents +it, and `multi-tenancy` is the tenant-predicate refinement — neither is +restated here. + ## Components - **Standard**: OWASP Top 10 (current edition) diff --git a/docs/website/content/concerns/twelve-factor.md b/docs/website/content/concerns/twelve-factor.md index 72c1bc14..9f2d4cfb 100644 --- a/docs/website/content/concerns/twelve-factor.md +++ b/docs/website/content/concerns/twelve-factor.md @@ -369,15 +369,11 @@ at the codebase, the release artifact, or the runbook and confirm or reject it. ## Stay in your lane (boundary with sibling concerns) -- Do **not** decide how many deployables the system ships as or where their - seams fall here — that is `deployment-topology`. This concern is the contract - **each** deployable honors. -- Do **not** specify log schema, span naming, trace context, or metric names - here — that is `o11y-otel`. This concern only says the event stream goes to - **stdout** and the process does not manage log files. -- Do **not** review cluster / Helm / image-build / secret-injection mechanics - here — that is `k8s-kind` / the `deploy-target` filler. This concern produces - the process the runtime consumes; it does not specify the runtime. +See `concern.md` for the canonical Boundary (vs `deployment-topology`, +`o11y-otel`, `k8s-kind` / the `deploy-target` filler). These practices are +the per-process operational contract — defer to the neighbor named there for +the deployable count and seams, the log/metric/trace schema, and the cluster +/ Helm / image-build mechanics. ## Artifact impact (what selecting this changes) diff --git a/docs/website/content/concerns/unity-catalog.md b/docs/website/content/concerns/unity-catalog.md index bff100fe..2ad478f1 100644 --- a/docs/website/content/concerns/unity-catalog.md +++ b/docs/website/content/concerns/unity-catalog.md @@ -27,30 +27,17 @@ data-modeling or access-control concern (see `## Boundary`). This concern owns **how data and AI assets are governed on Databricks** — the catalog namespace, the grant model, lineage, and governed external storage. It -is **Databricks' concrete realization** of data governance and must not be -restated as, or used to duplicate, the generic neighbors: - -- **Generic data modeling / `domain-driven-design`** owns *what the data - means* — entities, aggregates, invariants, the logical schema. Unity Catalog - owns *where those assets are registered and who may touch them* — the physical - `catalog.schema.object` namespace and the GRANT model over it. Model the - domain in the domain concern; register and govern it here. Do not re-derive - the logical model in this concern. -- **`security-owasp` / generic authz** owns *application-layer* authentication - and authorization (sessions, RBAC in app code, the OWASP risks). Unity Catalog - owns *data-layer* governance enforced by the platform at query time - (privileges, row filters, column masks, ownership). A Databricks App still - authenticates its users at the app layer; Unity Catalog is the additional - governance the *data* access flows through. Do not collapse the two — app - RBAC is not a substitute for catalog grants, and catalog grants are not a - substitute for app authz. -- **`databricks-declarative-pipelines`** *produces* governed datasets (its - streaming tables and materialized views land in a catalog/schema); Unity - Catalog *governs* them. The pipeline declares the dataset; this concern owns - the grants, ownership, and lineage on the result. -- **`databricks-apps`** *consumes* governed data; Unity Catalog *governs* the - consumption. This concern owns the rule that an app reads through Unity - Catalog grants, not around them. +is **Databricks' concrete realization** of data governance. + +For the auth family (where app-layer `authorization-model` and catalog grants +**compose** — neither substitutes for the other), see +[README-auth-family.md](../readme-auth-family/). For the logical domain +model, defer to `domain-driven-design`: model entities/aggregates there; +register and govern the physical `catalog.schema.object` namespace here. +`databricks-declarative-pipelines` *produces* governed datasets; this concern +owns the grants, ownership, and lineage on the result. `databricks-apps` +*consumes* governed data; this concern owns the rule that an app reads +through Unity Catalog grants, not around them. ## Components @@ -206,9 +193,10 @@ Agents working in any of these activities inherit the practices below via the be These practices govern **how data and AI assets are registered, granted, and lineage-tracked on Databricks**. They are the Databricks realization of data -governance; they do not restate the logical data model (`domain-driven-design`) -or application-layer authz (`security-owasp`) — see the boundary in -`concern.md`. +governance. For the boundary (composition with `authorization-model` / +`security-owasp`, `domain-driven-design`, `databricks-apps`, +`databricks-declarative-pipelines`) see `concern.md` and the auth family +ownership table at [README-auth-family.md](../readme-auth-family/). ## Requirements (Frame activity) @@ -259,19 +247,6 @@ or application-layer authz (`security-owasp`) — see the boundary in - Verify **lineage** is captured for the product's key tables (upstream → downstream visible in Unity Catalog lineage). -## Boundary with neighbors - -- **vs domain modeling (`domain-driven-design`)**: model entities/aggregates in - the domain concern; register and govern the physical namespace here. Do not - re-derive the logical model in catalog terms. -- **vs `security-owasp` / app authz**: app-layer auth and catalog grants - **compose**; neither substitutes for the other. App RBAC does not replace - `SELECT`/`MODIFY` grants; catalog grants do not replace app authentication. -- **vs `databricks-declarative-pipelines`**: the pipeline declares the dataset; - this concern owns the grants/ownership/lineage on the result. -- **vs `databricks-apps`**: the app consumes data; this concern owns the rule - that it consumes **through** Unity Catalog. - ## Quality Gates - All tables, views, volumes, and models the product uses are **registered in diff --git a/docs/website/content/concerns/ux-radix.md b/docs/website/content/concerns/ux-radix.md index 36af2ae9..8b973ff6 100644 --- a/docs/website/content/concerns/ux-radix.md +++ b/docs/website/content/concerns/ux-radix.md @@ -18,7 +18,12 @@ ui, frontend ## Components -- **Primitives**: Radix UI (headless, accessible by default) +- **Primitives**: Radix UI (headless, accessible by default) — this concern is + the canonical owner of the Radix prescription; other concerns reference here +- **Component library**: shadcn/ui (copied components, not an npm dependency) + built on top of Radix — this concern is the canonical owner of the shadcn + prescription; framework concerns (e.g. `react-nextjs`) reference here rather + than re-prescribing - **Patterns**: WAI-ARIA design patterns for all interactive widgets - **Scope**: Searching, editing, navigation, selection, and disclosure @@ -119,16 +124,21 @@ Any project with interactive user interfaces that involve searching, editing, navigating, or selecting data. Composes with `a11y-wcag-aa` for compliance requirements and `react-nextjs` for React-specific implementation patterns. Framework-agnostic in principle — the patterns are WAI-ARIA standards; Radix -is the reference implementation for React projects. +is the reference implementation for React projects, and shadcn/ui is the +prescribed component library that wraps Radix for React projects. ## Artifact Impact Selecting this concern requires these artifacts to change (a selected concern absent from them is drift): -- DESIGN_SYSTEM: WAI-ARIA interaction patterns for search/edit/nav/select/disclosure; Radix primitives; states +- ADR: shadcn/ui + Radix as the UI-primitives prescription (component library copied, not npm dep) +- DESIGN_SYSTEM: WAI-ARIA interaction patterns for search/edit/nav/select/disclosure; Radix primitives; shadcn component conventions; states - TEST_PLAN: keyboard navigation, focus-return, active-state, and live-filter behavior checks ## ADR References +- ADR-011: ux-radix owns the Radix and shadcn component-library prescription; + framework concerns reference rather than re-prescribe + ## Practices by activity Agents working in any of these activities inherit the practices below via the bead's context digest. @@ -209,6 +219,22 @@ carry every state: ## Implementation +### Component library (canonical owner) + +This concern is the canonical owner of the Radix and shadcn/ui prescription. +Framework concerns (e.g. `react-nextjs`) MUST reference this concern for UI +primitives instead of re-prescribing shadcn or Radix themselves. + +- Use **shadcn/ui** as the component library, built on Radix primitives. +- shadcn components are **copied into the project** (e.g. via + `npx shadcn@latest add `) and customized in `components/ui/` — + shadcn is NOT installed as an npm dependency. +- Customize copied shadcn components to match the project's design tokens; + keep the Radix-based accessibility behavior intact (do not override keyboard + or focus handlers with custom logic that breaks the WAI-ARIA contract). +- For headless behavior not covered by shadcn, use the underlying Radix + primitive directly. + ### Radix component mapping | Pattern | Radix Primitive | Notes | diff --git a/docs/website/content/research/databricks-platform-substitution.md b/docs/website/content/research/databricks-platform-substitution.md new file mode 100644 index 00000000..8c2c44e0 --- /dev/null +++ b/docs/website/content/research/databricks-platform-substitution.md @@ -0,0 +1,56 @@ +--- +title: "Databricks Platform Substitution Reference" +slug: databricks-platform-substitution +generated: true +--- +# Databricks Platform Substitution Reference + +If you are adopting HELIX data artifacts (`data-prd`, `data-architecture`, +`data-quality-expectations`) on a platform other than Databricks, substitute +the platform-specific terms below. The artifact templates and prompts stay +the same; only the named platform features change. + +## Catalog / Governance + +| Databricks Concept | Snowflake Equivalent | BigQuery Equivalent | On-Prem / Other | +|---|---|---|---| +| Unity Catalog (UC) hierarchy (metastore → catalog → schema → volume/table) | Database + Schema + Table | Project + Dataset + Table | Database + Schema | +| UC fine-grained access (row filters, column masks) | Row-access policies + masking policies | Authorized views + column-level security | Database views, RLS extensions | +| UC lineage | Object dependencies + Account Usage | Data lineage in Dataplex | Manual lineage in dbt/Marquez | + +## Ingestion / Streaming + +| Databricks Concept | Snowflake Equivalent | BigQuery Equivalent | On-Prem / Other | +|---|---|---|---| +| Auto Loader (incremental cloud-file ingestion) | Snowpipe or native connectors | Dataflow, BigQuery Connector Hub, Streaming Inserts | Apache NiFi, Kafka connectors | +| Streaming Tables (declarative streaming with EXPECT) | Stream-triggered materialized views + native checks | Dataflow with Beam assertions | Apache Flink with custom state management | +| Spark Structured Streaming | Snowflake streams + tasks | Dataflow | Apache Spark / Flink | + +## Orchestration / Pipelines + +| Databricks Concept | Snowflake Equivalent | BigQuery Equivalent | On-Prem / Other | +|---|---|---|---| +| Databricks Workflows / Jobs | Snowflake Tasks | Cloud Composer (Airflow) or Cloud Workflows | Apache Airflow, Dagster, dbt Cloud | +| Delta Live Tables / SDP pipeline orchestration | Dynamic Tables | Dataform | dbt, Dagster assets | + +## Quality / Contracts + +| Databricks Concept | Snowflake Equivalent | BigQuery Equivalent | On-Prem / Other | +|---|---|---|---| +| SDP `EXPECT ... ON VIOLATION ...` | Data Quality checks + Task error handling | BigQuery Data Quality API + Cloud Workflows | dbt tests, Great Expectations, custom assertions | +| SDP Genie test generation | dbt auto-generate tests from table samples | BigQuery Data Catalog insights | dbt, custom metadata scanning | +| Lakeview dashboards (quality monitoring) | Snowflake Dashboards | Looker, Data Studio | Grafana, custom dashboards | + +## Storage / Formats + +| Databricks Concept | Snowflake Equivalent | BigQuery Equivalent | On-Prem / Other | +|---|---|---|---| +| Delta Lake format | Iceberg or proprietary formats | Native BigQuery tables | Apache Parquet, Iceberg, Hudi | +| Medallion layers (Bronze / Silver / Gold) | Same pattern applies universally | Same pattern applies universally | Same pattern applies universally | + +## Compute / Cost + +| Databricks Concept | Snowflake Equivalent | BigQuery Equivalent | On-Prem / Other | +|---|---|---|---| +| DBU budget estimation | Credit consumption | On-demand or flat-rate pricing | Compute resource allocation | +| All-purpose / Jobs / Serverless SQL compute tiers | Warehouses (XS through 6XL) | On-demand vs reservation slots | Cluster sizing | diff --git a/justfile b/justfile index 3360affe..e93b4a0f 100644 --- a/justfile +++ b/justfile @@ -51,6 +51,10 @@ check-prose-redundancy: test-website-generated: bash tests/validate-website-generated.sh +# Validate artifact-type schemas (meta.yml required_sections <-> template.md H2s) +validate-artifact-schemas: + python3 scripts/helix_validate_artifact_meta.py + # Check internal links + images in a production-shaped build (with the /helix base path) check-website-links: cd website && hugo --gc --minify --baseURL "https://documentdrivendx.github.io/helix/" >/dev/null diff --git a/scripts/generate-reference.py b/scripts/generate-reference.py index 194cb814..b8d7580b 100644 --- a/scripts/generate-reference.py +++ b/scripts/generate-reference.py @@ -8,11 +8,14 @@ concerns from the upstream source-of-truth in `workflows/`. Reads: - workflows/activities//artifacts// -> meta.yml, dependencies.yaml, - prompt.md, template.md, - example.md (sometimes) + workflows/activities//artifacts// -> meta.yml, prompt.md, + template.md, example.md (sometimes) workflows/concerns// -> concern.md, practices.md +Dependencies (requires/enables/informs/referenced_by) come from `meta.yml` +under `dependencies.*` and `relationships.*`. See ADR-004 — there is no +separate `dependencies.yaml` file in the catalog. + Writes: docs/website/content/reference/glossary/artifacts/_index.md docs/website/content/reference/glossary/artifacts/.md (one per artifact) @@ -584,7 +587,6 @@ def collect_artifacts(): "activity": activity, "src_dir": art_dir, "meta": load_yaml(art_dir / "meta.yml"), - "deps": load_yaml(art_dir / "dependencies.yaml"), "prompt": load_text(art_dir / "prompt.md"), "template": load_text(art_dir / "template.md"), "example": load_text(art_dir / "example.md") if (art_dir / "example.md").exists() else None, @@ -596,26 +598,28 @@ def _safe_dict(value) -> dict: return value if isinstance(value, dict) else {} -def get_artifact_name(meta: dict, deps: dict, slug: str) -> str: +def get_artifact_name(meta: dict, slug: str) -> str: return ( _safe_dict(meta.get("artifact")).get("name") - or _safe_dict(deps.get("artifact")).get("name") or humanize(slug) ) -def get_artifact_description(meta: dict, deps: dict) -> str: +def get_artifact_description(meta: dict) -> str: desc = ( meta.get("description") - or deps.get("description") or _safe_dict(meta.get("artifact")).get("description") or "" ) return desc.strip() if isinstance(desc, str) else "" -def get_relationships(meta: dict, deps: dict) -> dict: - """Pull requires/enables/informs/referenced_by from deps + meta.""" +def get_relationships(meta: dict) -> dict: + """Pull requires/enables/informs/referenced_by from meta.yml. + + Per ADR-004, `meta.yml` is the single source of truth for artifact-type + dependencies — there is no separate `dependencies.yaml` file. + """ out = {"requires": [], "enables": [], "informs": [], "referenced_by": []} def _bucket_from(source: dict, key: str, name_field: str): @@ -635,11 +639,10 @@ def _bucket_from(source: dict, key: str, name_field: str): }) return items - # deps.yaml is more structured; try it first, fall back to meta.yml - out["requires"] = _bucket_from(deps, "requires", "input") or _bucket_from(meta, "requires", "input") - out["enables"] = _bucket_from(deps, "enables", "output") or _bucket_from(meta, "enables", "output") + out["requires"] = _bucket_from(meta, "requires", "input") + out["enables"] = _bucket_from(meta, "enables", "output") - rel = _safe_dict(deps.get("relationships")) or _safe_dict(meta.get("relationships")) + rel = _safe_dict(meta.get("relationships")) for slug in rel.get("informs", []) or []: if isinstance(slug, str): out["informs"].append({"slug": slug, "name": humanize(slug)}) @@ -650,8 +653,8 @@ def _bucket_from(source: dict, key: str, name_field: str): return out -def get_output_location(meta: dict, deps: dict) -> str: - out = _safe_dict(meta.get("output")) or _safe_dict(deps.get("output")) +def get_output_location(meta: dict) -> str: + out = _safe_dict(meta.get("output")) return (out.get("location") or "").strip() @@ -677,10 +680,10 @@ def render_relationship_list(items: list, all_slugs: set, slug_to_url: dict[str, def render_artifact_page(art: dict, all_slugs: set, slug_to_url: dict[str, str]) -> str: slug = art["slug"] activity = art["activity"] - name = get_artifact_name(art["meta"], art["deps"], slug) - desc = get_artifact_description(art["meta"], art["deps"]) - rels = get_relationships(art["meta"], art["deps"]) - output = get_output_location(art["meta"], art["deps"]) + name = get_artifact_name(art["meta"], slug) + desc = get_artifact_description(art["meta"]) + rels = get_relationships(art["meta"]) + output = get_output_location(art["meta"]) purpose = extract_markdown_section(art["prompt"], "Purpose") key_principles = extract_markdown_section(art["prompt"], "Key Principles") lane = extract_markdown_section(art["prompt"], "Stay in Your Lane") @@ -771,8 +774,8 @@ def append_artifact_cards( out.append("{{< cards >}}") for art in artifacts: slug = art["slug"] - name = get_artifact_name(art["meta"], art["deps"], slug) - desc = get_artifact_description(art["meta"], art["deps"]) or slug + name = get_artifact_name(art["meta"], slug) + desc = get_artifact_description(art["meta"]) or slug target = artifact_url(slug, slug_to_url) link = site_relurl(page_url, target) if page_url else target out.append( @@ -1025,7 +1028,7 @@ def render_concern_page(c: dict) -> str: if c["concern_md"]: out.append("## Description") out.append("") - out.append(strip_first_h1(c["concern_md"]).strip()) + out.append(rewrite_family_readme_links(strip_first_h1(c["concern_md"]).strip())) out.append("") if c["practices_md"]: @@ -1036,12 +1039,27 @@ def render_concern_page(c: dict) -> str: "via the bead's context digest." ) out.append("") - out.append(strip_first_h1(c["practices_md"]).strip()) + out.append(rewrite_family_readme_links(strip_first_h1(c["practices_md"]).strip())) out.append("") return "\n".join(out) +def rewrite_family_readme_links(md: str) -> str: + """Rewrite source-tree links like `../README-auth-family.md` to the Hugo URL. + + In `workflows/concerns//concern.md`, the family README sits at + `../README-auth-family.md`. The website mirror flattens concerns to + `concerns/.md`, and Hugo serves the README at `/concerns/readme-auth-family/`. + Rewrite so the published link points at the published page. + """ + return re.sub( + r"\]\(\.\./README-([a-z0-9-]+)\.md\)", + lambda m: f"](../readme-{m.group(1)}/)", + md, + ) + + def render_concerns_index(concerns: list) -> str: # Group by display category (consolidated) by_display: dict[str, list[dict]] = {} @@ -1157,6 +1175,23 @@ def main() -> None: for c in concerns: (CONCERNS_DEST / f"{c['slug']}.md").write_text(render_concern_page(c)) + # Family-level READMEs (e.g. README-auth-family.md) live alongside the + # per-concern directories in workflows/concerns/ and are referenced by + # `../README-auth-family.md` links from concern.md files. Publish them + # at the same flat level so those links resolve in Hugo. + for readme in sorted(CONCERNS_SRC.glob("README-*.md")): + body = load_text(readme) + title = "Auth family — ownership table" + # Strip first H1 to avoid duplicate-title leak + body = strip_first_h1(body) + frontmatter = ( + "---\n" + f"title: \"{title}\"\n" + "generated: true\n" + "---\n\n" + ) + (CONCERNS_DEST / readme.name).write_text(frontmatter + body) + # Concerns index (CONCERNS_DEST / "_index.md").write_text(render_concerns_index(concerns)) diff --git a/scripts/helix_validate_artifact_meta.py b/scripts/helix_validate_artifact_meta.py new file mode 100644 index 00000000..a584dbc7 --- /dev/null +++ b/scripts/helix_validate_artifact_meta.py @@ -0,0 +1,200 @@ +#!/usr/bin/env python3 +"""Validate artifact-type schemas: required_sections must match template H2s. + +For every artifact under workflows/activities//artifacts//, read +meta.yml and extract validation.required_sections (list of section IDs). Then +read template.md, extract its H2 headings, slugify each (lowercase, replace +runs of non-alphanumeric with single underscore, strip leading/trailing +underscores). Assert every required_section appears as an H2 slug. + +Stdlib-only: meta.yml is parsed by a small purpose-built reader that handles +the limited shape used by HELIX artifact meta files (mapping keys, list of +strings). This keeps the script runnable as `python3 scripts/...` with no +third-party dependency. + +Exit codes: 0 = all clean; 1 = drift detected. +""" +from __future__ import annotations + +import re +import sys +from pathlib import Path + + +REPO_ROOT = Path(__file__).resolve().parent.parent +ACTIVITIES_ROOT = REPO_ROOT / "workflows" / "activities" + + +def slugify(heading: str) -> str: + """Lowercase, collapse non-alphanumerics to underscores, trim underscores.""" + slug = re.sub(r"[^a-z0-9]+", "_", heading.lower()) + return slug.strip("_") + + +def extract_h2_slugs(template_path: Path) -> list[str]: + slugs: list[str] = [] + for line in template_path.read_text(encoding="utf-8").splitlines(): + m = re.match(r"^##\s+(?!#)(.+?)\s*$", line) + if m: + slugs.append(slugify(m.group(1))) + return slugs + + +def extract_output_format(meta_path: Path) -> str | None: + """Extract `output.format` from a HELIX meta.yml, or None if absent. + + The validator's H2-vs-required_sections check only applies to markdown + artifacts. YAML artifacts (metric-definition, etc.) declare their schema + by YAML keys, not template H2s, so they are skipped. + """ + text = meta_path.read_text(encoding="utf-8").splitlines() + in_output = False + for line in text: + if re.match(r"^output\s*:\s*$", line): + in_output = True + continue + if in_output: + if line and not line.startswith((" ", "\t")) and ":" in line: + return None + m = re.match(r"^\s+format\s*:\s*(.+?)\s*$", line) + if m: + value = m.group(1).strip() + if (value.startswith('"') and value.endswith('"')) or ( + value.startswith("'") and value.endswith("'") + ): + value = value[1:-1] + return value + return None + + +def extract_required_sections(meta_path: Path) -> list[str] | None: + """Parse out validation.required_sections from a HELIX meta.yml. + + Returns the list of section IDs (possibly empty), or None if no + `validation.required_sections` key is declared at all. We deliberately + keep this narrow: locate the `validation:` block, then within it the + `required_sections:` list, then read subsequent `- item` lines until the + indentation drops back out of the list. This matches every HELIX meta + file shape we care about without pulling in PyYAML. + """ + text = meta_path.read_text(encoding="utf-8").splitlines() + + # Find top-level `validation:` (column 0). + validation_line = None + for i, line in enumerate(text): + if re.match(r"^validation\s*:\s*$", line): + validation_line = i + break + if validation_line is None: + return None + + # Within the validation block, find `required_sections:` (any indent > 0). + required_line = None + for i in range(validation_line + 1, len(text)): + line = text[i] + # Block ends when we hit another top-level key. + if line and not line.startswith((" ", "\t")) and ":" in line: + break + m = re.match(r"^(\s+)required_sections\s*:\s*(.*)$", line) + if m: + required_line = i + break + if required_line is None: + return None + + # Collect list items (lines like ` - foo`) immediately following. + items: list[str] = [] + base_indent = None + for i in range(required_line + 1, len(text)): + line = text[i] + if not line.strip(): + continue + # Stop when we leave the list (a key at lower indent than items, or + # we hit a non-list line at the same indent that's not a `-` item). + stripped = line.lstrip(" \t") + indent = len(line) - len(stripped) + if not stripped.startswith("- "): + # Could be the next sibling key inside `validation:` — stop. + if base_indent is None or indent <= base_indent - 2: + break + # Or a continuation we don't model — stop conservatively. + break + if base_indent is None: + base_indent = indent + elif indent != base_indent: + break + value = stripped[2:].strip() + # Strip quotes if present. + if (value.startswith('"') and value.endswith('"')) or ( + value.startswith("'") and value.endswith("'") + ): + value = value[1:-1] + items.append(value) + + return items + + +def iter_artifact_dirs() -> list[Path]: + dirs: list[Path] = [] + if not ACTIVITIES_ROOT.is_dir(): + return dirs + for phase in sorted(ACTIVITIES_ROOT.iterdir()): + artifacts_dir = phase / "artifacts" + if not artifacts_dir.is_dir(): + continue + for art_dir in sorted(artifacts_dir.iterdir()): + if art_dir.is_dir() and (art_dir / "meta.yml").exists(): + dirs.append(art_dir) + return dirs + + +def validate(art_dir: Path) -> list[str]: + """Return a list of failure messages for this artifact (empty if clean).""" + meta_path = art_dir / "meta.yml" + template_path = art_dir / "template.md" + failures: list[str] = [] + + # Skip non-markdown artifacts: their schema lives in YAML keys, not H2s. + fmt = extract_output_format(meta_path) + if fmt and fmt.lower() != "markdown": + return failures + + required = extract_required_sections(meta_path) + if not required: + return failures # no required_sections declared, nothing to check + + if not template_path.exists(): + failures.append( + f"{art_dir.relative_to(REPO_ROOT)}: meta.yml declares required_sections " + f"but template.md is missing" + ) + return failures + + h2_slugs = set(extract_h2_slugs(template_path)) + for section_id in required: + if section_id not in h2_slugs: + failures.append( + f"{art_dir.relative_to(REPO_ROOT)}: required_section '{section_id}' " + f"not found as H2 in template.md (have: {sorted(h2_slugs)})" + ) + return failures + + +def main() -> int: + art_dirs = iter_artifact_dirs() + all_failures: list[str] = [] + for art_dir in art_dirs: + all_failures.extend(validate(art_dir)) + + if all_failures: + for msg in all_failures: + print(f"FAIL: {msg}") + print(f"\nFAIL: {len(all_failures)} drift(s) across {len(art_dirs)} artifact type(s)") + return 1 + + print(f"OK: {len(art_dirs)} artifact type(s) validated (required_sections <-> template H2s)") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tests/validate-website-generated.sh b/tests/validate-website-generated.sh index 0729788c..e352bf9d 100755 --- a/tests/validate-website-generated.sh +++ b/tests/validate-website-generated.sh @@ -38,7 +38,9 @@ fi # Coverage signal: every upstream concern and artifact-type has a rendered page. src_concerns=$(find workflows/concerns -mindepth 1 -maxdepth 1 -type d | wc -l | tr -d ' ') -out_concerns=$(find "$content/concerns" -name '*.md' ! -name '_index.md' | wc -l | tr -d ' ') +# Family READMEs (e.g. README-auth-family.md) are published alongside concerns +# but are not concerns themselves; exclude them from the coverage count. +out_concerns=$(find "$content/concerns" -name '*.md' ! -name '_index.md' ! -name 'README-*.md' | wc -l | tr -d ' ') src_arttypes=$(find workflows/activities/*/artifacts -mindepth 1 -maxdepth 1 -type d | wc -l | tr -d ' ') out_arttypes=$(find "$content/artifact-types" -name '*.md' ! -name '_index.md' | wc -l | tr -d ' ') diff --git a/website/e2e/microsite.spec.ts b/website/e2e/microsite.spec.ts index 7776256d..9e0988b6 100644 --- a/website/e2e/microsite.spec.ts +++ b/website/e2e/microsite.spec.ts @@ -59,7 +59,7 @@ test.describe('Why HELIX', () => { test('principles page covers all eight', async ({ page }) => { await page.goto('/why/principles/') await expect(page.getByRole('heading', { name: /1\.\s+Planning and execution/ })).toBeVisible() - await expect(page.getByRole('heading', { name: /3\.\s+Authority order/ })).toBeVisible() + await expect(page.getByRole('heading', { name: /3\.\s+The artifact authority hierarchy/ })).toBeVisible() await expect(page.getByRole('heading', { name: /6\.\s+Autonomy is supervised/ })).toBeVisible() await expect(page.getByRole('heading', { name: /8\.\s+Least power wins/ })).toBeVisible() }) @@ -210,7 +210,7 @@ test.describe('Reference', () => { test('concepts page covers core ideas', async ({ page }) => { await page.goto('/reference/glossary/concepts/') - await expect(page.getByRole('heading', { name: 'Authority Order' })).toBeVisible() + await expect(page.getByRole('heading', { name: 'Artifact Authority Hierarchy' })).toBeVisible() await expect(page.getByRole('heading', { name: 'Context Digest' })).toBeVisible() }) diff --git a/workflows/actions/frame.md b/workflows/actions/frame.md index dfdf663b..902f6aa6 100644 --- a/workflows/actions/frame.md +++ b/workflows/actions/frame.md @@ -246,10 +246,12 @@ If you drafted or updated user stories, also check: ### Validation gate -After refinement, read `dependencies.yaml` and `prompt.md` from each artifact -directory you touched. Verify all **blocking** quality checks pass. If any -fail, revise the artifact before proceeding to Step 4. Do not commit an -artifact that fails a blocking check. +After refinement, read `meta.yml` (specifically the `validation.quality_checks` +section) and `prompt.md` from each artifact directory you touched. Verify all +**blocking** quality checks pass. If any fail, revise the artifact before +proceeding to Step 4. Do not commit an artifact that fails a blocking check. +See ADR-004 for why validation rules live in `meta.yml` rather than a separate +`dependencies.yaml`. ## STEP 3.5 — Principles Bootstrap (if needed) @@ -287,8 +289,8 @@ See the measure action for the full pattern. 1. **Artifact completeness**: All required artifacts for the scope have been created or updated. -2. **Validation gates**: All blocking quality checks from `dependencies.yaml` - and `prompt.md` pass for each artifact. +2. **Validation gates**: All blocking quality checks from `meta.yml` + (`validation.quality_checks`) and `prompt.md` pass for each artifact. 3. **Work item creation**: Downstream design work items have been filed for each feature spec. 4. **Concern selection (required)**: verify concern selection was performed — diff --git a/workflows/activities/00-discover/artifacts/opportunity-canvas/meta.yml b/workflows/activities/00-discover/artifacts/opportunity-canvas/meta.yml index c850e5f0..ff296a07 100644 --- a/workflows/activities/00-discover/artifacts/opportunity-canvas/meta.yml +++ b/workflows/activities/00-discover/artifacts/opportunity-canvas/meta.yml @@ -22,7 +22,7 @@ validation: - solution_concept - key_metrics - unfair_advantage - - go_nogo_checklist + - go_no_go_decision quality_checks: - check: problem_validated diff --git a/workflows/activities/00-discover/artifacts/product-vision/meta.yml b/workflows/activities/00-discover/artifacts/product-vision/meta.yml index 79cad244..660735bd 100644 --- a/workflows/activities/00-discover/artifacts/product-vision/meta.yml +++ b/workflows/activities/00-discover/artifacts/product-vision/meta.yml @@ -22,7 +22,7 @@ validation: - vision - user_experience - target_market - - value_propositions + - key_value_propositions - success_definition - why_now diff --git a/workflows/activities/01-frame/README.md b/workflows/activities/01-frame/README.md index ec3ee026..47e47391 100644 --- a/workflows/activities/01-frame/README.md +++ b/workflows/activities/01-frame/README.md @@ -31,7 +31,8 @@ Criteria to proceed to Design activity (defined in `GATE.yaml`): The Frame activity produces four core artifacts. Each artifact directory under `artifacts/` includes `template.md` (structure), `prompt.md` (section-by-section -guidance and quality checklist), and `dependencies.yaml` (validation rules). +guidance and quality checklist), and `meta.yml` (validation rules and +type-level dependencies; see ADR-004). ### 1. Product Requirements Document (PRD) diff --git a/workflows/activities/01-frame/artifacts/compliance-requirements/meta.yml b/workflows/activities/01-frame/artifacts/compliance-requirements/meta.yml index 187d2c45..5f2f5f4a 100644 --- a/workflows/activities/01-frame/artifacts/compliance-requirements/meta.yml +++ b/workflows/activities/01-frame/artifacts/compliance-requirements/meta.yml @@ -17,9 +17,9 @@ output: validation: required_sections: - - regulatory_landscape_analysis + - applicable_regulations - compliance_requirements_matrix - - data_classification + - data_classification_and_handling - implementation_plan quality_checks: diff --git a/workflows/activities/01-frame/artifacts/compliance-requirements/prompt.md b/workflows/activities/01-frame/artifacts/compliance-requirements/prompt.md index 573590d9..f4dd2d67 100644 --- a/workflows/activities/01-frame/artifacts/compliance-requirements/prompt.md +++ b/workflows/activities/01-frame/artifacts/compliance-requirements/prompt.md @@ -1,5 +1,14 @@ # Compliance Requirements Analysis Prompt -Document the compliance obligations for this project in the local template. +Map the external regulations and standards that apply to this project to the +controls and evidence that satisfy them. + +## Traceability chain + +This artifact's place in the security triangle: **regulations -> controls**. +- See `security-requirements` for the testable acceptance criteria that + exercise those controls. +- See `threat-model` for the abuse paths those controls mitigate and the + STRIDE owners. ## Reference Anchors @@ -13,18 +22,16 @@ Use these local resource summaries as grounding: ## Focus - Identify only the regulations and standards that actually apply to this system. - Mark uncertain applicability explicitly and route it to counsel or compliance review. -- Map each obligation to its source, affected scope, concrete controls, owners, evidence, and timing. +- Map each obligation to its source, affected scope, the control(s) that satisfy + it, owners, evidence, and timing. - Keep the result concise and implementation-relevant. - Do not invent legal conclusions. State assumptions and review gaps. - -## Role Boundary - -Compliance Requirements is not Security Requirements or the Threat Model. It -defines external obligations and the controls/evidence needed to satisfy them. -Security Requirements turns those obligations into security behavior; Threat -Model analyzes abuse paths and mitigations. +- Do not enumerate threats, attacker behavior, or STRIDE categories. Cross- + reference `threat-model` for those. +- Do not author testable acceptance criteria for controls. Cross-reference + `security-requirements` for that. ## Completion Criteria - Applicable, not-applicable, and uncertain obligations are identified. -- Controls, evidence, owners, and deadlines are explicit. +- Each applicable obligation names the control(s), evidence, owner, and deadline. - No generic filler is added. diff --git a/workflows/activities/01-frame/artifacts/data-prd/example.md b/workflows/activities/01-frame/artifacts/data-prd/example.md deleted file mode 100644 index 846322bf..00000000 --- a/workflows/activities/01-frame/artifacts/data-prd/example.md +++ /dev/null @@ -1,165 +0,0 @@ ---- -ddx: - id: example.data-prd.customer-360 ---- - -# Data Product Requirements Document: Customer-360 Analytics - -## Summary - -Customer-360 is a unified analytics dataset that reconciles Salesforce customer -accounts with Stripe payment history and customer lifecycle events. It powers -sales forecasting, churn analysis, and account health scoring. The first release -ingests 12 months of historical data and supports daily incremental updates for -ongoing revenue and subscription trends. Success means sales analysts can query a -single Gold table to answer "total ARR per customer," "months since last payment," -and "invoice aging" without joining across multiple systems. - -## Problem and Goals - -### Problem - -Sales and finance teams use Salesforce for CRM and Stripe for payment processing, -but data is siloed. Analysts manually pull exports, reconcile customer IDs across -systems, and build fragile spreadsheets to answer cash-flow and churn questions. -Each reconciliation takes 3-4 hours and risks stale data by the time insights are -shared. - -### Goals - -1. Unify Salesforce accounts and Stripe customers under a single customer identity. -2. Preserve payment lineage: which invoice, subscription, or charge event drove - each revenue transaction. -3. Deliver Gold tables that require no post-query transformation for common - business questions. - -### Success Metrics - -| Metric | Target | Measurement Method | -|--------|--------|--------------------| -| Query response time | Under 5 seconds for customer-year aggregations | Databricks query execution time | -| Data freshness | Daily updates complete before 7am UTC | Pipeline job logs and SLA monitoring | -| Customer ID reconciliation accuracy | 98% of Salesforce accounts matched to Stripe customers | Manual audit sample of matched pairs | -| Analyst adoption | 10+ scheduled reports consuming Gold tables | Databricks workspace audit logs | - -### Non-Goals - -- Streaming ingestion (batch-daily is sufficient for v1). -- Custom churn-scoring algorithms (featurized data enables external ML). -- Integration with Marketo, HubSpot, or third-party data warehouses. -- Supporting sub-account hierarchies within Stripe. - -## Data Consumers - -| Role | Team | Key Questions | Tables | -|------|------|---------------|--------| -| Sales Analyst | Revenue | ARR by account, churn risk score, invoice aging | dim_customer, fct_monthly_revenue | -| Finance Manager | Revenue Operations | Subscription changes, failed payment count, billing exceptions | fct_subscription_event, fct_payment_transaction | -| Data Engineer | Analytics | Lineage validation, late-arriving facts, reconciliation record counts | all layer tables, metadata.processing_log | - -## Data Sources - -| Source System | Entity | Volume (12-month history) | Frequency | SLA | -|---------------|--------|---------------------------|-----------|-----| -| Salesforce | Account records (customer name, owner, industry) | 5,000 accounts | Daily export (API) | 6-hour lag acceptable | -| Salesforce | Opportunity records (pipeline, closed deals) | 50,000 opps | Daily export (API) | 6-hour lag acceptable | -| Stripe | Customer objects (email, metadata tags) | 4,500 customers | Real-time webhook (v1 batch daily) | 24-hour batch acceptable for v1 | -| Stripe | Subscription objects (plan, status, start/end dates) | 6,200 subscriptions | Real-time webhook (v1 batch daily) | 24-hour batch acceptable for v1 | -| Stripe | Invoice records (amount, status, line items) | 45,000 invoices | Real-time webhook (v1 batch daily) | 24-hour batch acceptable for v1 | -| Stripe | Charge records (card, amount, outcome) | 180,000 charges | Real-time webhook (v1 batch daily) | 24-hour batch acceptable for v1 | - -## Data Quality Requirements - -### Coverage - -| Expectation | Threshold | Owner | Severity | -|-------------|-----------|-------|----------| -| Salesforce account export completeness | ≥ 95% of prior day's account count | Data Eng | P0 (block load) | -| Stripe customer export completeness | ≥ 95% of prior day's distinct customers | Data Eng | P0 (block load) | -| Customer ID reconciliation rate | ≥ 98% of matched Salesforce-Stripe pairs | Data Eng | P1 (alert, allow backfill) | -| Invoice line-item accuracy | 100% of invoices ≥ $0; amount matches sum of line items | Data Eng | P0 (block load) | - -### Freshness - -| Dataset | Expected latency | Acceptable Delay | Monitoring | -|---------|------------------|------------------|------------| -| Bronze Salesforce (raw export) | Within 6 hours of API call | 8 hours | Query SLA dashboard | -| Bronze Stripe (raw export) | Within 24 hours of event (batch v1) | 26 hours | Query SLA dashboard | -| Silver (deduplicated, reconciled) | Within 8 hours of Bronze completion | 10 hours | Pipeline job history | -| Gold (aggregated facts) | Before 7am UTC daily | 8am UTC | Scheduled report execution | - -### Metadata Requirements - -- **Lineage**: Every Gold row must record the source Salesforce and Stripe record IDs, - import batch ID, and last-modified timestamp. -- **Data Quality Flags**: Silver layer must include reconciliation match confidence, - late-arriving fact flags, and anomaly detection scores. -- **Processing Metadata**: All tables must include `_loaded_at`, `_processed_by_job_id`, - and `_source_system`. - -## Databricks-Specific Technical Context - -### Workspace and Catalog - -- **Catalog**: main (Unity Catalog enabled) -- **Schema Naming**: customer_360_{bronze,silver,gold} -- **Medallion Layers**: Separate schemas for isolation and access control - -### Compute - -- **Orchestration**: Databricks Workflows with daily 10pm UTC trigger -- **Compute**: Job cluster, 2 workers, 8 DBU/hour estimate -- **Language**: Python with PySpark for orchestration, SQL for transformations - -### Access Control - -- **Bronze**: Data Engineer read/write; Analyst no access (sensitive raw PII) -- **Silver**: Data Engineer read/write; Analyst read-only (internal team only) -- **Gold**: Data Engineer read/write; Analyst read; Managers read (published via BI) - -### Compliance - -- **PII Handling**: Customer email and phone hashed in Silver+ layers; raw email in Bronze only -- **Retention**: Bronze/Silver retained 90 days; Gold retained 3 years -- **Masking**: Stripe card tokens never stored; card brand only in Silver - -## Constraints and Assumptions - -### Constraints - -- **Data Volume**: 12-month history = ~250GB uncompressed; daily increments ~1GB -- **Cost**: ≤ $500 USD/month DBU spend for batch jobs + queries -- **Latency**: Daily batch only for v1; no streaming/real-time SLAs - -### Assumptions - -- Salesforce and Stripe customer data stabilizes within 24 hours of creation -- No single customer appears as multiple accounts in Salesforce (will validate in reconciliation) -- Email is a sufficient match key for Salesforce-Stripe reconciliation (vs. manual linking) - -### Dependencies - -- Salesforce API credentials and rate-limit quota -- Stripe API credentials and historical export access -- Databricks Unity Catalog workspace with ≥ 2 worker nodes available - -## Risks - -| Risk | Probability | Impact | Mitigation | -|------|-------------|--------|------------| -| Email format inconsistency blocks Stripe-Salesforce joins | High | High | Implement fuzzy email matching (lowercase, domain normalization) in Silver | -| Stripe subscription deletion removes invoice lineage | Medium | High | Copy foreign keys to Bronze before join; archive deleted subscriptions in reference table | -| Gold table query cost exceeds budget | Medium | Medium | Add partition pruning by month; set query-timeout alarms; publish consumption dashboard | - -## Open Questions - -- [ ] Does Salesforce contain sub-accounts or partner-account hierarchies? Will this affect the 1:1 customer assumption? -- [ ] Are there known Stripe test customers in the export that should be filtered in Bronze? -- [ ] Which BI tool will query Gold tables (Tableau, Looker, Sigma)? Will it handle incremental refresh? - -## Success Criteria - -Customer-360 is successful when: -1. Analysts execute fewer than 5 manual export-and-reconcile cycles per quarter. -2. Sales team's monthly revenue forecast uses only Gold table queries (no spreadsheets). -3. Churn alerting dashboard refreshes successfully every morning by 7am UTC for 30 consecutive days. diff --git a/workflows/activities/01-frame/artifacts/data-prd/meta.yml b/workflows/activities/01-frame/artifacts/data-prd/meta.yml deleted file mode 100644 index a6dc9a39..00000000 --- a/workflows/activities/01-frame/artifacts/data-prd/meta.yml +++ /dev/null @@ -1,165 +0,0 @@ -# Metadata for Data Product Requirements Document (Data PRD) - -artifact: - name: Data Product Requirements Document - id: data-prd - type: document - activity: frame - -description: | - Data-scoped authority document that translates data vision into prioritized, - measurable requirements, data quality standards, data consumer expectations, - and Databricks platform context. Supplements or replaces the general PRD when - the primary artifact is a data product (pipeline, warehouse, data platform, or - data service). - -output: - location: docs/helix/01-frame/data-prd.md - format: markdown - naming: data-prd.md - examples: - - data-prd.md - -validation: - required_sections: - - executive_summary - - problem_statement - - goals_and_objectives - - data_sources - - data_consumers - - requirements_overview - - data_quality_requirements - - databricks_context - - success_metrics - - risks_and_mitigation - - quality_checks: - - check: metrics_quantified - description: All success metrics must have numeric targets (throughput, latency, freshness, quality score) - severity: blocking - - - check: data_consumers_identified - description: Data consumers table must name actual roles, systems, or teams with their use cases - severity: blocking - - - check: requirements_prioritized - description: Requirements must be categorized as P0/P1/P2 - severity: blocking - - - check: quality_thresholds_defined - description: Data quality requirements must specify acceptable thresholds (completeness %, null %, freshness SLA) - severity: blocking - - - check: sources_documented - description: Data sources table must list source system, schema, update frequency, and quality baseline - severity: warning - - - check: databricks_specifics_complete - description: Databricks context must name catalog, schema, medallion layer, and compute tier assumptions - severity: warning - - automated_checks: - - pattern: "\\[NEEDS CLARIFICATION\\]|\\[TODO\\]|\\[TBD\\]" - expected: 0 - message: Remove all clarification markers before approval - - - pattern: "P0|Must Have" - expected: ">0" - message: Data PRD must define at least one P0 requirement - -variables: - - name: data_product_name - description: Name of the data product or pipeline - required: true - - - name: primary_consumers - description: Teams, systems, or roles that consume this data - required: true - - - name: source_systems - description: Upstream systems providing data (Salesforce, Stripe, APIs, databases) - required: true - - - name: freshness_sla - description: Expected data freshness (e.g., "hourly", "daily", "real-time") - required: false - - - name: catalog_schema - description: Target Databricks catalog and schema (e.g., "prod.customer_360") - required: false - -prompts: - generation: prompt.md - review: | - Review this Data PRD for: - 1. Clarity of the data problem and business context - 2. Specific data consumers and their use cases - 3. Quantified success metrics (throughput, latency, quality) - 4. Data quality requirements with thresholds - 5. Databricks platform assumptions (catalog, schema, medallion layer) - 6. Completeness of source system inventory - 7. Identification and mitigation of data risks - -template: - file: template.md - sections: - - executive_summary: "Write this last; summarize data product purpose and impact" - - problem_statement: "Start here; describe the business problem the data product solves" - - data_sources: "List upstream systems, schemas, and refresh cadence" - - data_consumers: "Name the teams/systems that depend on this data and their use cases" - - requirements_overview: "Organized by P0/P1/P2 priority" - - data_quality_requirements: "Table with quality dimensions, thresholds, and enforcement" - - databricks_context: "Target catalog, schema, medallion layer, compute strategy" - -examples: - - file: example.md - description: Customer 360 data product PRD with Salesforce/Stripe sources, medallion architecture, and SLA requirements - -relationships: - informed_by: - - product-vision - - informs: - - data-architecture - - feature-specification - - user-stories - - referenced_by: - - data-architecture - - technical-design - -references: - - title: "Databricks Unity Catalog Documentation" - file: docs/resources/databricks-unity-catalog.md - relationship: "Grounds catalog, schema, and access control governance for data products" - - - title: "Databricks Medallion Architecture Pattern" - file: docs/resources/databricks-medallion-architecture.md - relationship: "Grounds Bronze/Silver/Gold layer strategy and data transformation semantics" - - - title: "Databricks Semantic Layer and Delta Live Tables" - file: docs/resources/databricks-delta-live-tables.md - relationship: "Grounds declarative data quality expectations and pipeline orchestration" - - - title: "Data Mesh and Domain-Driven Data Platform Design" - file: docs/resources/data-mesh.md - relationship: "Grounds data product thinking and data consumer accountability" - -workflow: - creation_order: 1 - review_cycles: 2-3 - approval_required: true - approvers: - - Data Product Owner - - Data Engineering Lead - - Key Data Consumer Stakeholder - -tags: - - data-engineering - - data-product - - requirements - - databricks - - quality - -version: 1.0.0 -last_updated: 2026-05-18 diff --git a/workflows/activities/01-frame/artifacts/data-prd/prompt.md b/workflows/activities/01-frame/artifacts/data-prd/prompt.md deleted file mode 100644 index e56661a3..00000000 --- a/workflows/activities/01-frame/artifacts/data-prd/prompt.md +++ /dev/null @@ -1,72 +0,0 @@ -# Data Product Requirements (Data PRD) Generation Prompt - -Create a Data PRD that frames the data product problem, scope, quality requirements, -and success criteria clearly enough that downstream data architecture, quality -expectations, and implementation work can trace back to it. - -## Purpose - -The Data PRD is the **data-product-scope authority for what data to build and why**. -Its unique job is to translate business intent into data-centric requirements: -data sources, consumer personas, quality contracts, technical constraints (catalog, -schema, medallion layer), and measurable success metrics. It sits between the -general Product Vision and Data Architecture. Every data pipeline design choice -and quality expectation should trace back to a Data PRD requirement. - -## Reference Anchors - -Use these local resource summaries as grounding: - -- `docs/resources/databricks-unity-catalog.md` grounds data governance through - unified catalog hierarchies (metastore → catalog → schema → volume/table). -- `docs/resources/databricks-lakehouse-medallion-architecture.md` grounds - medallion topology (Bronze/Silver/Gold) and layer responsibilities in a - Lakehouse. -- `docs/resources/databricks-sdp.md` grounds Databricks Semantic Data Platform - governance, lineage, and quality contracts through `EXPECT ... ON VIOLATION ...` - clauses and SDP-aware pipeline patterns. - -## Focus - -- Name the data sources (internal and external), ingestion cadence, and expected - volume/velocity. -- Define data consumers and their use cases: who consumes the data, what decisions - they make, and what quality they depend on. -- List data quality requirements as testable contracts: completeness, accuracy, - freshness, schema consistency, referential integrity. -- Specify the Databricks technical context: target catalog/schema, medallion - layer (Bronze/Silver/Gold), pipeline type (Auto Loader, Streaming Tables, SQL - pipeline), and estimated DBU budget. -- Frame success metrics for the data product itself: SLA compliance (freshness, - availability), consumer satisfaction, cost per GB, defect escape rate. - -## Role Boundary - -Data PRD is not a general product PRD, data model, pipeline design, or quality -implementation. It specifies *what* data requirements are, not *how* to implement -them. - -**Databricks Platform Substitution:** If you are adopting this on another data -platform, substitute as follows: - -| Databricks Concept | Snowflake Equivalent | BigQuery Equivalent | On-Prem / Other | -|---|---|---|---| -| Unity Catalog (UC) hierarchy | Database + Schema + Table | Project + Dataset + Table | Database + Schema | -| Medallion architecture (Bronze/Silver/Gold) | Same pattern applies universally | Same pattern applies universally | Same pattern applies universally | -| Auto Loader, Streaming Tables | Snowpipe, Stream-triggered tasks | Dataflow, BigQuery Streaming Inserts | Apache Spark Structured Streaming, Airflow | -| SDP `EXPECT ... ON VIOLATION ...` | Data Quality checks in Snowflake | BigQuery Data Quality API | dbt tests, Great Expectations assertions | -| DBU budget estimation | Credit consumption | On-demand or flat-rate pricing | Compute resource allocation | - -## Completion Criteria - -- Data sources table names each external/internal source, ingestion pattern, and - freshness requirement. -- Data consumers table includes consumer role, use case, and quality SLA. -- Quality requirements are specific constraints (not aspirational); each one is - testable in Data Quality Expectations. -- Technical context names the target catalog, schema, medallion layer, and - pipeline type. -- Success metrics have numeric targets and measurement methods (e.g., "SLA - compliance > 95% measured by on-time delivery vs. promised refresh cadence"). -- Requirements trace upward to the Product Vision or general PRD and downward - to Data Architecture and Data Quality Expectations. diff --git a/workflows/activities/01-frame/artifacts/data-prd/template.md b/workflows/activities/01-frame/artifacts/data-prd/template.md deleted file mode 100644 index 08c252e8..00000000 --- a/workflows/activities/01-frame/artifacts/data-prd/template.md +++ /dev/null @@ -1,163 +0,0 @@ ---- -ddx: - id: data-prd ---- - -# Data Product Requirements Document - -## Executive Summary - -[This section works as a standalone 1-pager. Include: what data product we are building, who uses it, what business problem it solves, the data solution approach, and the top 2-3 success metrics. Write this last — it should be a distillation of the full PRD, not an introduction. Someone reading only this section should understand the data product well enough to decide whether to read the rest.] - -## Problem and Goals - -### Problem Statement - -[What is broken or missing in the current data landscape? Who is affected? Be specific: not "users struggle with reporting" but "sales analysts spend 4 hours per week reconciling pipeline outputs with source systems because current freshness is 24 hours and source data changes hourly."] - -### Business Goals - -1. [Primary goal — what changes for the organization or data consumers] -2. [Secondary goal — secondary business impact or risk reduction] - -### Goals and Objectives - -| Goal | Objective | Success Criteria | -|------|-----------|------------------| -| [Goal] | [How we achieve it via data] | [Measurable outcome] | - -### Non-Goals - -[What we are explicitly not trying to achieve. Each non-goal should exclude something a reasonable person might assume is in scope.] - -## Data Consumers - -### Primary Consumer: [Name/Role] - -**Team**: [Data Engineering, Analytics, Product, Finance, etc.] -**Use Case**: [What do they do with this data? How does it change their workflow?] -**Frequency**: [Real-time, daily, weekly, ad-hoc?] -**Key Tables/Feeds**: [Which outputs matter most to this consumer?] - -### Secondary Consumer: [Name/Role] - -[Repeat above structure for each secondary consumer.] - -### Data Consumer Requirements Table - -| Consumer | Use Case | Freshness SLA | Latency Tolerance | Key Dimensions | Access Level | -|----------|----------|---------------|-------------------|----------------|--------------| -| [Team] | [What they do] | [e.g., hourly] | [max delay] | [customer_id, product_id, ...] | [Row-level, Column-level, or Full] | - -## Data Sources - -### Source System Inventory - -| Source System | Schema / Table | Owner | Update Frequency | Quality Baseline | Notes | -|---------------|----------------|-------|------------------|------------------|-------| -| [e.g., Salesforce] | [e.g., Accounts, Opportunities] | [Team] | [hourly, daily, on-demand] | [% completeness, freshness] | [Data model version, API limits, retry policy] | - -## Requirements Overview - -### P0 (Must Have) - -[Critical requirements blocking data product delivery or violating SLA.] - -- Requirement: [e.g., "Ingest Salesforce data within 1 hour of transaction close"] -- Requirement: [e.g., "Support daily rollup of customer spend by product category"] - -### P1 (Should Have) - -[Important for end-user value or operational stability.] - -- Requirement: [e.g., "Detect and alert on data quality anomalies within 15 minutes"] -- Requirement: [e.g., "Enable drill-down to daily granularity without latency penalty"] - -### P2 (Nice to Have) - -[Nice-to-have enhancements deferred if time-constrained.] - -- Requirement: [e.g., "Publish historical snapshots for year-over-year comparison"] - -## Data Quality Requirements - -### Quality Dimensions and Thresholds - -| Dimension | P0 Threshold | P1 Threshold | Measurement Method | Enforcement | -|-----------|--------------|--------------|-------------------|-------------| -| Completeness | [e.g., ≥99%] | [e.g., ≥95%] | [Count NULLs / total rows] | [Alert if falls below P0] | -| Timeliness | [e.g., ≤1 hour lag] | [e.g., ≤4 hour lag] | [MAX(ingestion_time) - MAX(source_time)] | [Reject data if exceeds P0] | -| Accuracy | [e.g., ≥98% match to source] | [e.g., ≥95% match] | [Row-count reconciliation + sample audit] | [Manual review + auto-reject if P0 fails] | -| Uniqueness | [e.g., PK has no duplicates] | [as P0] | [COUNT(*) = COUNT(DISTINCT PK)] | [Fail ingestion] | - -### Data Quality Expectations - -[Reference the [[data-quality-expectations]] document for detailed EXPECT clauses per medallion layer. This section summarizes which quality dimensions are critical.] - -## Databricks-Specific Technical Context - -### Catalog and Schema Strategy - -- **Target Catalog**: [e.g., `prod`, `analytics`, or domain-specific catalog] -- **Target Schema**: [e.g., `customer_360`, `payment_events`] -- **Medallion Layers**: Bronze (raw), Silver (validated), Gold (business) -- **Access Control Model**: [UC policies, Row-level security, column masking] - -### Medallion Layer Strategy - -- **Bronze**: [Raw data ingestion point; what is ingested, what constraints apply, how often] -- **Silver**: [Validated and deduplicated; transformation rules, quality gates] -- **Gold**: [Business-ready tables; aggregations, dimensions, fact tables for consumers] - -### Databricks Features and Compute Strategy - -| Feature | Decision | Rationale | -|---------|----------|-----------| -| Ingestion Pattern | [Auto Loader, Streaming Tables, batch] | [Why this choice?] | -| Processing Model | [Streaming, Batch, Incremental] | [Freshness SLA and cost tradeoff] | -| Compute Tier | [All-purpose, Jobs, Serverless] | [Workload characteristics, cost model] | -| Storage Format | [Delta, Parquet, CSV] | [Durability, query performance needs] | -| DBU Budget (Monthly) | [Estimated spend] | [Based on row volume, freshness, complexity] | - -### Governance and Compliance - -- **Data Classification**: [Public, Internal, Sensitive, PII] -- **Retention Policy**: [e.g., Bronze: 7 days, Silver: 90 days, Gold: 2 years] -- **Audit Trail**: [Who accessed what, when, why] -- **Lineage Tracking**: [Table-to-table dependencies for impact analysis] - -## Success Metrics - -| Metric | Target | Baseline | Measurement Method | Cadence | -|--------|--------|----------|-------------------|---------| -| [Throughput] | [e.g., 1M rows/day] | [Current: 100K rows/day] | [COUNT(*) from production table] | Daily | -| [Latency] | [e.g., ≤1 hour end-to-end] | [Current: 4 hours] | [MAX(ingestion_timestamp) - MAX(source_timestamp)] | Hourly | -| [Quality Score] | [e.g., ≥98%] | [Current: 85%] | [Automated quality checks pass rate] | Daily | -| [Cost per GB] | [e.g., $0.05/GB/month] | [Current: $0.12/GB/month] | [DBU spend / data volume] | Monthly | - -## Risks and Mitigation - -| Risk | Likelihood | Impact | Mitigation Strategy | -|------|------------|--------|-------------------| -| [e.g., Source API rate limits] | [High] | [Data ingestion delays] | [Implement backoff + queue; alert on throttling] | -| [e.g., Schema drift in source] | [Medium] | [Pipeline failure or silent corruption] | [Schema registry; alerts on new columns; manual review] | -| [e.g., Cost overrun from compute] | [Medium] | [Budget overrun] | [Set DBU limits; implement cost monitoring] | - ---- - -## Review Checklist - -Use this checklist during review to validate that the data PRD is complete and ready for design: - -- [ ] **Executive Summary** is self-contained and clearly states the data product purpose -- [ ] **Problem Statement** is specific: includes failure mode, frequency, and affected stakeholders (not generic) -- [ ] **Data Consumers table** names actual teams/roles with concrete use cases (not abstract personas) -- [ ] **Data Sources** table is complete: every source system has an entry with owner and update frequency -- [ ] **Requirements** are prioritized (P0/P1/P2) and traceable to data consumer use cases -- [ ] **Quality Dimensions** have numeric thresholds (not vague targets like "high quality") -- [ ] **Databricks Context** specifies catalog, schema, medallion layer strategy, and compute approach -- [ ] **Success Metrics** are quantified: throughput (rows/day), latency (max age), quality score (%), cost ($/GB) -- [ ] **Risks** are specific with likelihood/impact assessment and mitigation tactics -- [ ] No `[TBD]`, `[TODO]`, or `[NEEDS CLARIFICATION]` markers remain -- [ ] Naming and terminology align with Databricks Unity Catalog and SDP conventions -- [ ] Every P0 requirement has at least one data quality expectation (link to [[data-quality-expectations]]) diff --git a/workflows/activities/01-frame/artifacts/feasibility-study/meta.yml b/workflows/activities/01-frame/artifacts/feasibility-study/meta.yml index d6cac7b2..ea4b44e9 100644 --- a/workflows/activities/01-frame/artifacts/feasibility-study/meta.yml +++ b/workflows/activities/01-frame/artifacts/feasibility-study/meta.yml @@ -19,13 +19,11 @@ output: validation: required_sections: - executive_summary - - technical_feasibility - - business_feasibility - - operational_feasibility - - resource_feasibility - - risk_assessment - - recommendations + - feasibility_assessment + - risks + - alternatives - decision_framework + - next_steps quality_checks: - check: all_dimensions_assessed diff --git a/workflows/activities/01-frame/artifacts/feasibility-study/template.md b/workflows/activities/01-frame/artifacts/feasibility-study/template.md index fea5dde0..948cb3b2 100644 --- a/workflows/activities/01-frame/artifacts/feasibility-study/template.md +++ b/workflows/activities/01-frame/artifacts/feasibility-study/template.md @@ -65,6 +65,16 @@ ddx: - **Feasibility**: [Brief] - **Decision**: Carry forward | Reject +## Decision Framework + +| Criterion | Status | Rationale | +|-----------|--------|-----------| +| Technical buildability | Pass / Risk / Fail | [Rationale] | +| Business value | Pass / Risk / Fail | [Rationale] | +| Operational supportability | Pass / Risk / Fail | [Rationale] | +| Compliance readiness | Pass / Risk / Fail | [Rationale] | +| Resource availability | Pass / Risk / Fail | [Rationale] | + ## Next Steps 1. [Action] 2. [Action] diff --git a/workflows/activities/01-frame/artifacts/feature-registry/prompt.md b/workflows/activities/01-frame/artifacts/feature-registry/prompt.md index 9afad1a8..890d32b3 100644 --- a/workflows/activities/01-frame/artifacts/feature-registry/prompt.md +++ b/workflows/activities/01-frame/artifacts/feature-registry/prompt.md @@ -27,3 +27,51 @@ requirements; Feature Specifications define behavior; runtime work items track e - IDs are unique and never reused. - The registry stays easy to scan. - Every active feature links to its governing artifact or clearly states the missing link. + +## Promotion from Parking Lot + +Per [ADR-010](../../../../../docs/helix/02-design/adr/ADR-010-feature-registry-parking-lot-handoff.md), +`feature-registry` and `parking-lot` stay separate and the handoff is an +explicit recorded transition. When a parking-lot entry's revisit criteria are +met, promote it to the registry with the following procedure. + +### Promotion Criteria + +A parking-lot entry is eligible for promotion when all of these hold: + +- **Revisit trigger fired**: the objective condition recorded on the entry has + occurred (date reached, dependency landed, external signal observed). +- **Scope decided**: the item has been re-scoped into something a feature spec + can be written against — not a vague idea kept warm. +- **Owner assigned**: a named owner accepts responsibility for the feature + through at least the `specified` status. +- **Blocking ADRs resolved**: any ADR that the entry was waiting on has landed + (accepted or rejected); items still pending ADRs stay parked. +- **Dependencies available**: prerequisite features listed on the entry are at + a status that unblocks this one (typically `built` or later). + +If any criterion fails, leave the entry parked and update the rationale or +revisit trigger to reflect what is still missing. + +### Promotion Procedure + +1. **Assign the next sequential FEAT-XXX**: never reuse an ID, including IDs + from cancelled or deprecated features. Add the new row to `Active Features` + with initial status `Draft` (or `Specified` if the spec is ready to land in + the same change). +2. **Record the back-link to the parking-lot source**: in the new feature row's + `Source` column, cite the parking-lot entry title (e.g. + `parking-lot:`). This makes the parked-to-active transition + auditable. +3. **Update the parking-lot entry**: mark the entry as promoted, record the + assigned `FEAT-XXX`, and the promotion date. Do not delete the parking-lot + entry — the historical record is part of the back-link. +4. **Seed traceability**: populate the new feature's `Trace Links` row with the + feature spec, stories, designs, tests, and release placeholders. Empty cells + are fine; missing cells are not. +5. **Carry over dependencies**: copy any `Dependencies` from the parking-lot + entry into the registry's `Dependencies` table, expressed as FEAT-to-FEAT + edges where the prerequisites have FEAT-XXX IDs. + +The promotion is complete when the new `FEAT-XXX` row exists with a back-link, +the parking-lot entry records the promotion, and traceability rows are seeded. diff --git a/workflows/activities/01-frame/artifacts/feature-specification/meta.yml b/workflows/activities/01-frame/artifacts/feature-specification/meta.yml index add34f79..321d01ca 100644 --- a/workflows/activities/01-frame/artifacts/feature-specification/meta.yml +++ b/workflows/activities/01-frame/artifacts/feature-specification/meta.yml @@ -9,9 +9,11 @@ artifact: description: | Feature-level authority document that translates PRD requirements into precise behavior, boundaries, functional areas, non-functional expectations, - acceptance criteria, edge cases, and feature-specific success measures. Broad - product-surface, workflow, IA, and documentation features must state the - ideal future state and map functional areas before requirements. + edge cases, and feature-specific success measures. Acceptance criteria live + in user-stories (ADR-009); the feature spec carries FR-IDs and the + decomposition test but does not define AC. Broad product-surface, workflow, + IA, and documentation features must state the ideal future state and map + functional areas before requirements. output: location: docs/helix/01-frame/features/ @@ -22,12 +24,10 @@ output: validation: required_sections: - problem_statement - - functional_requirements - - acceptance_criteria - - non_functional_requirements - - edge_cases + - requirements + - edge_cases_and_error_handling - success_metrics - - constraints + - constraints_and_assumptions - out_of_scope quality_checks: @@ -35,10 +35,6 @@ validation: description: All requirements must be testable severity: blocking - - check: acceptance_criteria_observable - description: Acceptance criteria must describe observable behavior, not implementation steps - severity: blocking - - check: future_state_first description: Broad feature specs must define the desired future state before optimizing around current problems severity: blocking diff --git a/workflows/activities/01-frame/artifacts/feature-specification/prompt.md b/workflows/activities/01-frame/artifacts/feature-specification/prompt.md index 1d0a7436..3ed248c0 100644 --- a/workflows/activities/01-frame/artifacts/feature-specification/prompt.md +++ b/workflows/activities/01-frame/artifacts/feature-specification/prompt.md @@ -1,7 +1,9 @@ # Feature Specification Generation Prompt Create a feature specification that is precise enough to support design, -user story creation, and test planning. +user story creation, and test planning. The feature spec owns FR-IDs, +functional areas, and the decomposition test — acceptance criteria live in +user-stories (see ADR-009) and must not be restated here. ## Storage Location @@ -11,8 +13,8 @@ Store at: `docs/helix/01-frame/features/FEAT-NNN-.md` A feature spec is the **feature-level authority for behavior and boundaries**. It translates PRD requirements into precise feature behavior, functional areas, -acceptance criteria, non-functional expectations, edge cases, and -feature-specific success measures. +non-functional expectations, edge cases, and feature-specific success measures. +Acceptance criteria belong to user stories (ADR-009) and are not defined here. It sits between the PRD (which defines product scope) and user stories (which define vertical slices through the feature). The feature spec owns feature @@ -43,8 +45,8 @@ but leaves no trace here is drift (reconcile-alignment Concern->Artifact Realiza explains why the change is needed; it should not be the only organizing frame. - **Scope, not solution** — describe what the feature must do, not how to build it. Implementation details belong in design docs. -- **Behavior, not journey** — specify feature behavior and acceptance criteria. - Put end-to-end user flow narrative in user stories. +- **Behavior, not journey** — specify feature behavior and boundaries. + Put end-to-end user flow narrative and acceptance criteria in user stories. - **One feature, one capability** — a feature spec covers exactly one capability (≈ one PRD subsystem). If it covers two, split it; apply the Decomposition test below to decide. A functional *area* is a sub-part of one capability, not a @@ -63,9 +65,6 @@ but leaves no trace here is drift (reconcile-alignment Concern->Artifact Realiza content. Stories are separate files with their own lifecycle. - **Testable requirements** — every functional requirement should be verifiable. If you can't describe how to test it, it's too vague. -- **Concrete acceptance examples** — add examples for important rules, - permissions, errors, and edge cases. They should show observable behavior, - not internal steps. - **Leave unknowns explicit** — use Open Questions at the bottom rather than inventing detail you don't have. @@ -74,8 +73,8 @@ but leaves no trace here is drift (reconcile-alignment Concern->Artifact Realiza | If you are writing... | Put it in... | |---|---| | Product goals, personas, launch priority, or product-level metrics | PRD | -| Feature behavior, boundaries, acceptance criteria, and edge cases | Feature Specification | -| A vertical user journey through one or more feature requirements | User Story | +| Feature behavior, boundaries, and edge cases | Feature Specification | +| A vertical user journey through one or more feature requirements, with acceptance criteria | User Story | | Component choices, data model, APIs, or implementation approach | Solution/Technical Design | | Detailed test cases, fixtures, or automation strategy | Test Plan or Story Test Plan | | Build sequencing and work slices | Implementation Plan | @@ -159,12 +158,6 @@ must do — user stories describe how users interact with these capabilities. If a requirement mentions two areas joined by "and", split it unless the relationship between those areas is itself the requirement. -### Acceptance Criteria -Capture observable examples for the highest-risk or most important -requirements. Use concise Given/When/Then phrasing if it helps, but do not -require Cucumber tooling. Each example should identify the requirement it -validates and the expected result. - ### Non-Functional Requirements Every NFR needs a specific target. "Must be fast" is not a requirement. "95th percentile response under 200ms" is. Only include NFRs relevant to @@ -207,7 +200,6 @@ committing. - [ ] Similar domain objects are separated before requirements are written - [ ] Functional requirements are grouped by area when a flat list would mix unrelated scopes - [ ] Every functional requirement is testable -- [ ] Acceptance criteria cover the highest-risk requirements with observable examples - [ ] Non-functional requirements have specific numeric targets - [ ] User stories are referenced by ID (not duplicated inline) - [ ] Dependencies name specific feature IDs and external systems diff --git a/workflows/activities/01-frame/artifacts/feature-specification/template.md b/workflows/activities/01-frame/artifacts/feature-specification/template.md index bf8c17cd..8f5346b3 100644 --- a/workflows/activities/01-frame/artifacts/feature-specification/template.md +++ b/workflows/activities/01-frame/artifacts/feature-specification/template.md @@ -62,16 +62,6 @@ clear, such as `HOME-01`, `TYPE-01`, `NAV-01`, or `FR-01` for narrow features.] [PREFIX-01]. [Requirement] [PREFIX-02]. [Requirement] -### Acceptance Criteria - -[Capture observable examples for the highest-risk or most important -requirements. Given/When/Then phrasing is allowed but not required. Do not -describe implementation steps.] - -| Requirement | Scenario | Given | When | Then | -|-------------|----------|-------|------|------| -| [PREFIX-01] | [Observable case] | [Starting state] | [User/system action] | [Expected result] | - ### Non-Functional Requirements - **Performance**: [Specific target, e.g., "95th percentile response < 200ms"] @@ -130,7 +120,7 @@ Use this checklist when reviewing a feature specification: - [ ] Requirements are grouped by functional area when a flat list would mix unrelated scopes - [ ] Domain objects that sound similar are explicitly separated (for example, artifact instances vs artifact types) - [ ] Every functional requirement is testable — you can write an assertion for it -- [ ] Acceptance criteria cover important happy paths, errors, and edge cases with observable outcomes +- [ ] Acceptance criteria are defined in the user stories that decompose this feature, not here (ADR-009) - [ ] Non-functional requirements have specific numeric targets, not "must be fast" - [ ] Edge cases cover realistic failure scenarios, not just happy paths - [ ] Success metrics are specific to this feature, not product-level metrics diff --git a/workflows/activities/01-frame/artifacts/parking-lot/prompt.md b/workflows/activities/01-frame/artifacts/parking-lot/prompt.md index 4b717221..87278ccf 100644 --- a/workflows/activities/01-frame/artifacts/parking-lot/prompt.md +++ b/workflows/activities/01-frame/artifacts/parking-lot/prompt.md @@ -25,3 +25,10 @@ or cancelled, not parked forever. - Deferred items are easy to find later. - Nothing active is buried here. - Every item has source, rationale, owner, and revisit trigger. + +## Promotion to Feature Registry + +When a parked entry's revisit trigger fires and the item is ready to enter the +active pipeline, follow the promotion procedure documented in the +[Feature Registry prompt](../feature-registry/prompt.md#promotion-from-parking-lot) +(per [ADR-010](../../../../../docs/helix/02-design/adr/ADR-010-feature-registry-parking-lot-handoff.md)). diff --git a/workflows/activities/01-frame/artifacts/parking-lot/template.md b/workflows/activities/01-frame/artifacts/parking-lot/template.md index 37d5a562..3b118a16 100644 --- a/workflows/activities/01-frame/artifacts/parking-lot/template.md +++ b/workflows/activities/01-frame/artifacts/parking-lot/template.md @@ -5,6 +5,11 @@ ddx: # Parking Lot (Deferred / Future Work) +## Purpose + +[Why this parking lot exists and what it covers — scope of deferred / future +work tracked here without distorting current commitments.] + ## Policy - Rejected items do not belong here; close or cancel them instead. - Active work does not belong here; track it in the Feature Registry and DDx. diff --git a/workflows/activities/01-frame/artifacts/pr-faq/meta.yml b/workflows/activities/01-frame/artifacts/pr-faq/meta.yml index 5f22852e..ddffdf5e 100644 --- a/workflows/activities/01-frame/artifacts/pr-faq/meta.yml +++ b/workflows/activities/01-frame/artifacts/pr-faq/meta.yml @@ -17,22 +17,9 @@ output: validation: required_sections: + - press_release - internal_product_argument - - core_thesis - - mechanism - - quality_model - - decision_autonomy_boundary - - press_release_headline - - press_release_subhead - - press_release_summary - - the_problem - - the_solution - - leader_quote - - how_it_works - - customer_quote - - availability - - external_faqs - - internal_faqs + - faq - downstream_projection quality_checks: diff --git a/workflows/activities/01-frame/artifacts/prd/meta.yml b/workflows/activities/01-frame/artifacts/prd/meta.yml index 027d0535..aa21bab6 100644 --- a/workflows/activities/01-frame/artifacts/prd/meta.yml +++ b/workflows/activities/01-frame/artifacts/prd/meta.yml @@ -9,7 +9,22 @@ artifact: description: | Product-scope authority document that translates vision into prioritized, measurable requirements, success metrics, non-goals, assumptions, and - acceptance sketches. + acceptance sketches. The `kind` field selects the framing variant: the + default `product` variant frames general product requirements; the `data` + variant frames a data product (pipeline, warehouse, data platform, or data + service) with data sources, consumers, quality contracts, and platform + context. The shape of the artifact is unified; the framing is parameterized. + +kind: + enum: + - product + - data + default: product + description: | + Framing variant for this PRD. `product` (default) frames general product + requirements. `data` frames a data product with data-source, consumer, + quality-contract, and platform-context guidance. New variants require an + ADR (see ADR-008). output: location: docs/helix/01-frame/prd.md diff --git a/workflows/activities/01-frame/artifacts/prd/prompt.md b/workflows/activities/01-frame/artifacts/prd/prompt.md index d02dcfa6..7e007976 100644 --- a/workflows/activities/01-frame/artifacts/prd/prompt.md +++ b/workflows/activities/01-frame/artifacts/prd/prompt.md @@ -4,9 +4,23 @@ Create a PRD that frames the problem, product scope, priorities, and success criteria clearly enough that downstream feature specs, designs, tests, and implementation work can trace back to it. +## Variant Selection (`kind`) + +The PRD has two framing variants selected by the `kind` field in `meta.yml`: + +- **`kind: product`** (default) — frames general product requirements. +- **`kind: data`** — frames a data product (pipeline, warehouse, data + platform, or data service) with data sources, consumers, quality + contracts, and platform context. + +The shape of the artifact is unified; sections marked **(kind: data)** below +swap in the data-product framing when `kind: data`. Per ADR-008, both +framings share one role and one template — pick the variant that matches the +authored artifact and follow its conditional guidance. + ## Storage Location -Store at: `docs/helix/01-frame/prd.md` +Store at: `docs/helix/01-frame/prd.md` (for either variant). ## Purpose @@ -16,6 +30,14 @@ requirements and boundaries. It sits between the product vision (which defines direction) and feature specs (which define feature-level detail). Every design decision and implementation choice should trace back to a PRD requirement. +**(kind: data)** When `kind: data`, the PRD is the **data-product-scope +authority for what data to build and why**. Its job is to translate business +intent into data-centric requirements: data sources, consumer personas, +quality contracts, technical constraints (catalog, schema, medallion layer), +and measurable success metrics. It sits between the Product Vision and the +data-architecture artifact. Every data pipeline design choice and quality +expectation should trace back to a Data PRD requirement. + ## Reference Anchors Use these local resource summaries as grounding: @@ -24,6 +46,18 @@ Use these local resource summaries as grounding: - `docs/resources/aha-prd-template.md` supports concise cross-functional scope: what is being built, who it is for, and how it delivers value. - `docs/resources/ibm-requirements-management.md` grounds measurable, prioritized, traceable requirements and validation/verification discipline. +**(kind: data)** When `kind: data`, also use: + +- `docs/resources/databricks-unity-catalog.md` grounds data governance through unified catalog hierarchies (metastore → catalog → schema → volume/table). +- `docs/resources/databricks-lakehouse-medallion-architecture.md` grounds medallion topology (Bronze/Silver/Gold) and layer responsibilities in a Lakehouse. +- `docs/resources/databricks-sdp.md` grounds Databricks Semantic Data Platform governance, lineage, and quality contracts through `EXPECT ... ON VIOLATION ...` clauses and SDP-aware pipeline patterns. + +If you adopt this on another data platform, substitute Databricks concepts +with the platform equivalent (Snowflake DB/Schema/Table; BigQuery +Project/Dataset/Table; Snowpipe/Streaming Inserts for Auto Loader; +dbt/Great Expectations for SDP `EXPECT` clauses). The medallion pattern +applies universally. + ## Key Principles - **Problem first** — the problem section should make someone feel the pain @@ -78,6 +112,13 @@ how you'll measure it. "User satisfaction" is not a metric. "NPS > 40 from monthly survey of active users" is. Drop the Timeline column — success metrics should define what success looks like, not when it happens. +**(kind: data)** When `kind: data`, frame metrics for the data product +itself: throughput (rows/day), latency (max age end-to-end), quality score +(percentage of expectations passing), cost per GB or per DBU, freshness-SLA +compliance, and consumer satisfaction. Each metric still needs a numeric +target and a named measurement method — e.g., "SLA compliance > 95% measured +by on-time delivery vs. promised refresh cadence." + ### Non-Goals Each non-goal should prevent scope creep on something plausible. "Not a general-purpose AI" is only useful if someone might think it is. Test: would @@ -89,6 +130,16 @@ Name them. Give them a role, goals, and pain points specific enough to validate with a real person. "Alex the Developer" with generic goals is a template, not a persona. +**(kind: data)** When `kind: data`, frame this section as **Data Consumers** +instead of personas. Name actual teams, systems, or roles that consume the +data, their use case (what decisions they make), their freshness/latency SLA, +the key dimensions they query, and their access level (row, column, full). +Add an inventory of upstream **Data Sources** in a parallel section: source +system, schema/table, owner, update frequency, quality baseline, and notes +on API limits or retry policy. Generic personas are insufficient for a data +product — the consumer and source tables drive every downstream design and +quality decision. + ### Requirements (P0/P1/P2) P0 = the product is broken without this. P1 = the product is weak without this. P2 = the product is better with this. If you have more than 7 P0s, @@ -103,6 +154,17 @@ scope. These are the detailed behavioral specs. Each one should be testable — someone reading it should know how to write an acceptance test. +**(kind: data)** When `kind: data`, the functional requirements describe +**data behavior**: ingestion cadence, deduplication rules, transformation +contracts, freshness windows, schema-evolution policy, and consumer-facing +table or feed definitions. Add a **Data Quality Requirements** subsection +with quality dimensions (completeness, timeliness, accuracy, uniqueness) +each carrying a P0 threshold, a P1 threshold, a measurement method, and an +enforcement strategy (alert, reject, quarantine). Reference the +`data-quality-expectations` artifact for executable `EXPECT` clauses per +medallion layer; the PRD owns the requirement, the expectations artifact +owns the contract. + **Group requirements under named subsystems.** Organize FRs by subsystem (not by priority) under canonical, parseable headings: `### Subsystem: `. Each `FR-n` belongs to **exactly one** subsystem. A subsystem is a cohesive capability @@ -142,6 +204,17 @@ you're documenting a choice that doesn't have an ADR yet, note it in Open Questions. If an existing ADR contradicts what you'd write here, the ADR governs until it's superseded. +**(kind: data)** When `kind: data`, frame the technical context as the +**data-platform context**: target catalog and schema (e.g., `prod.customer_360`), +medallion layer strategy (Bronze/Silver/Gold scopes and responsibilities), +ingestion pattern (Auto Loader, Streaming Tables, batch), processing model +(streaming vs batch vs incremental), compute tier (all-purpose, jobs, +serverless), storage format (Delta, Parquet), DBU budget assumption, and +governance posture (data classification, retention policy, audit trail, +lineage). Pin the access-control model: row-level security, column masking, +and the catalog policies that enforce it. Same ADR discipline applies — the +PRD records platform decisions, ADRs justify them. + ### Constraints Name real constraints, not aspirational ones. "Must work on mobile" is a constraint only if you'd otherwise skip it. Budget, compliance, and platform diff --git a/workflows/activities/01-frame/artifacts/prd/template.md b/workflows/activities/01-frame/artifacts/prd/template.md index f149e943..5950265e 100644 --- a/workflows/activities/01-frame/artifacts/prd/template.md +++ b/workflows/activities/01-frame/artifacts/prd/template.md @@ -1,17 +1,29 @@ --- ddx: id: prd +kind: product # `product` (default) frames general product requirements; `data` frames a data product (pipeline, warehouse, data platform, or service). See ADR-008. --- # Product Requirements Document +> **Variant guidance.** This template carries both the default `product` +> framing and a `data` framing. Sections marked **(kind: data)** apply when +> `kind: data` and replace the corresponding `product` framing above them. +> When `kind: product`, ignore the **(kind: data)** blocks. The shape of the +> document is the same; the framing is parameterized. + ## Summary [This section should work as a standalone 1-pager. Include: what we're building, who uses it, what problem it solves, the solution approach, and the top 2-3 success metrics. Write this last — it should be a distillation of the full PRD, not an introduction. Someone who reads only this section should -understand the product well enough to decide whether to read the rest.] +understand the product well enough to decide whether to read the rest. + +**(kind: data)** Frame this as a standalone 1-pager for the **data product**: +what data we are building, who consumes it, the business problem it solves, +the data solution approach (sources, medallion layer strategy, consumption +shape), and the top 2-3 success metrics (freshness, quality, cost).] ## Problem and Goals @@ -19,7 +31,12 @@ understand the product well enough to decide whether to read the rest.] [What is broken or missing? Who is affected? Be specific about the failure mode — not "users struggle with X" but "users spend N hours per week doing X -because Y doesn't exist."] +because Y doesn't exist." + +**(kind: data)** Be specific about the data failure mode — not "users +struggle with reporting" but "sales analysts spend 4 hours per week +reconciling pipeline outputs with source systems because current freshness +is 24 hours and source data changes hourly."] ### Goals @@ -32,6 +49,17 @@ because Y doesn't exist."] |--------|--------|--------------------| | [Metric] | [Numeric target] | [Named tool or process] | +**(kind: data)** When `kind: data`, frame metrics for the data product +itself (throughput, latency, quality score, cost per GB). Include a baseline +and cadence column: + +| Metric | Target | Baseline | Measurement Method | Cadence | +|--------|--------|----------|--------------------|---------| +| [Throughput] | [e.g., 1M rows/day] | [Current: 100K rows/day] | [COUNT(*) from production table] | Daily | +| [Latency] | [e.g., ≤1 hour end-to-end] | [Current: 4 hours] | [MAX(ingestion_timestamp) - MAX(source_timestamp)] | Hourly | +| [Quality Score] | [e.g., ≥98%] | [Current: 85%] | [Automated quality checks pass rate] | Daily | +| [Cost per GB] | [e.g., $0.05/GB/month] | [Current: $0.12/GB/month] | [DBU spend / data volume] | Monthly | + ### Non-Goals [What we are explicitly not trying to achieve. Each non-goal should exclude @@ -51,6 +79,31 @@ Deferred items tracked in `docs/helix/parking-lot.md`. [Same structure] +### (kind: data) Data Consumers + +[When `kind: data`, replace the persona blocks with concrete data consumers.] + +#### Primary Consumer: [Name/Role] + +**Team**: [Data Engineering, Analytics, Product, Finance, etc.] +**Use Case**: [What they do with the data; what decision it informs] +**Frequency**: [Real-time, daily, weekly, ad-hoc] +**Key Tables/Feeds**: [Which outputs matter most] + +#### Data Consumer Requirements Table + +| Consumer | Use Case | Freshness SLA | Latency Tolerance | Key Dimensions | Access Level | +|----------|----------|---------------|-------------------|----------------|--------------| +| [Team] | [What they do] | [e.g., hourly] | [max delay] | [customer_id, product_id, ...] | [Row-level, Column-level, or Full] | + +### (kind: data) Data Sources + +[Inventory of upstream systems supplying this data product.] + +| Source System | Schema / Table | Owner | Update Frequency | Quality Baseline | Notes | +|---------------|----------------|-------|------------------|------------------|-------| +| [e.g., Salesforce] | [e.g., Accounts, Opportunities] | [Team] | [hourly, daily, on-demand] | [% completeness, freshness] | [Data model version, API limits, retry policy] | + ## Requirements Each requirement should trace to the Product Vision and be specific enough to @@ -92,6 +145,19 @@ them sequentially; do not renumber on edit.] - **FR-3** — [behavioral requirement, testable] +### (kind: data) Data Quality Requirements + +[When `kind: data`, add this subsection. Quality dimensions with numeric +thresholds and enforcement strategy. Reference `data-quality-expectations` +for executable `EXPECT` clauses per medallion layer.] + +| Dimension | P0 Threshold | P1 Threshold | Measurement Method | Enforcement | +|-----------|--------------|--------------|--------------------|-------------| +| Completeness | [e.g., ≥99%] | [e.g., ≥95%] | [Count NULLs / total rows] | [Alert if below P0] | +| Timeliness | [e.g., ≤1 hour lag] | [e.g., ≤4 hour lag] | [MAX(ingestion_time) - MAX(source_time)] | [Reject data if exceeds P0] | +| Accuracy | [e.g., ≥98% match to source] | [e.g., ≥95% match] | [Row-count reconciliation + sample audit] | [Manual review + auto-reject if P0 fails] | +| Uniqueness | [e.g., PK has no duplicates] | [as P0] | [COUNT(*) = COUNT(DISTINCT PK)] | [Fail ingestion] | + ## Acceptance Test Sketches [For each P0 requirement, describe a concrete scenario with inputs and @@ -116,6 +182,28 @@ here isn't backed by an ADR yet, note it in Open Questions.] - **APIs**: [e.g., OpenAPI spec at docs/api/v2.yaml] - **Platform Targets**: [e.g., Linux, macOS; browser: Chrome/Firefox/Safari latest] +### (kind: data) Data Platform Context + +[When `kind: data`, replace the stack list above with platform context.] + +- **Target Catalog**: [e.g., `prod`, `analytics`, or domain-specific catalog] +- **Target Schema**: [e.g., `customer_360`, `payment_events`] +- **Medallion Layers**: Bronze (raw), Silver (validated), Gold (business) +- **Access Control Model**: [UC policies, row-level security, column masking] + +| Feature | Decision | Rationale | +|---------|----------|-----------| +| Ingestion Pattern | [Auto Loader, Streaming Tables, batch] | [Why this choice?] | +| Processing Model | [Streaming, Batch, Incremental] | [Freshness SLA and cost tradeoff] | +| Compute Tier | [All-purpose, Jobs, Serverless] | [Workload characteristics, cost model] | +| Storage Format | [Delta, Parquet, CSV] | [Durability, query performance needs] | +| DBU Budget (Monthly) | [Estimated spend] | [Based on row volume, freshness, complexity] | + +- **Data Classification**: [Public, Internal, Sensitive, PII] +- **Retention Policy**: [e.g., Bronze: 7 days, Silver: 90 days, Gold: 2 years] +- **Audit Trail**: [Who accessed what, when, why] +- **Lineage Tracking**: [Table-to-table dependencies for impact analysis] + ## Constraints, Assumptions, Dependencies ### Constraints diff --git a/workflows/activities/01-frame/artifacts/research-plan/meta.yml b/workflows/activities/01-frame/artifacts/research-plan/meta.yml index 27d59a5b..a11c478e 100644 --- a/workflows/activities/01-frame/artifacts/research-plan/meta.yml +++ b/workflows/activities/01-frame/artifacts/research-plan/meta.yml @@ -19,13 +19,10 @@ output: validation: required_sections: - research_objectives - - knowledge_gaps - research_methods - - success_criteria + - completion_criteria - timeline - - resource_requirements - - risk_assessment - - expected_outcomes + - research_risks quality_checks: - check: objectives_measurable diff --git a/workflows/activities/01-frame/artifacts/security-requirements/meta.yml b/workflows/activities/01-frame/artifacts/security-requirements/meta.yml index 15efa489..05ed29fb 100644 --- a/workflows/activities/01-frame/artifacts/security-requirements/meta.yml +++ b/workflows/activities/01-frame/artifacts/security-requirements/meta.yml @@ -17,10 +17,9 @@ output: validation: required_sections: - - security_user_stories + - required_controls - compliance_requirements - - risk_assessment - - acceptance_criteria + - security_risks quality_checks: - check: all_requirements_have_acceptance_criteria diff --git a/workflows/activities/01-frame/artifacts/security-requirements/prompt.md b/workflows/activities/01-frame/artifacts/security-requirements/prompt.md index c1b0a969..24a29284 100644 --- a/workflows/activities/01-frame/artifacts/security-requirements/prompt.md +++ b/workflows/activities/01-frame/artifacts/security-requirements/prompt.md @@ -1,5 +1,14 @@ # Security Requirements Generation Prompt -Document the security requirements the project must satisfy before design and build. +Specify the testable acceptance criteria for the security controls the project +must satisfy before design and build. + +## Traceability chain + +This artifact's place in the security triangle: **controls -> testable +acceptance criteria**. +- See `compliance-requirements` for the regulations that motivate each control. +- See `threat-model` for the STRIDE categories and mitigation owners those + controls answer to. ## Reference Anchors @@ -14,19 +23,18 @@ Use these local resource summaries as grounding: ## Focus - Cover authentication, authorization, data protection, privacy, validation, and logging. -- Turn requirements into concrete controls and tests where possible. -- Keep compliance and risk notes brief but explicit. -- Make every requirement verifiable by design review, automated test, manual test, or audit evidence. - -## Role Boundary - -Security Requirements is not the Threat Model or Security Architecture. It -states what security outcomes and controls must be true. Threat Model explains -how the system can be attacked; Security Architecture explains where and how -controls are implemented. +- Express each requirement as a testable acceptance criterion (design review, + automated test, manual test, or audit evidence). +- For each control, cross-reference the originating obligation in + `compliance-requirements` and the STRIDE owner in `threat-model`. +- Do not restate regulatory text or applicability analysis. Cross-reference + `compliance-requirements`. +- Do not enumerate attacker behavior, data flows, or STRIDE categories. + Cross-reference `threat-model`. ## Completion Criteria -- Required controls are identified. -- Risks and assumptions are visible. +- Required controls are identified and each has at least one testable + acceptance criterion. +- Each criterion traces to a compliance obligation, a threat-model STRIDE + owner, or both. - The result is specific enough to guide design. -- Acceptance criteria are testable and traceable to controls, risks, or compliance obligations. diff --git a/workflows/activities/01-frame/artifacts/stakeholder-map/meta.yml b/workflows/activities/01-frame/artifacts/stakeholder-map/meta.yml index 97bdea14..bfb91729 100644 --- a/workflows/activities/01-frame/artifacts/stakeholder-map/meta.yml +++ b/workflows/activities/01-frame/artifacts/stakeholder-map/meta.yml @@ -18,12 +18,11 @@ output: validation: required_sections: - - stakeholder_identification + - primary_stakeholders_high_influence_high_interest + - secondary_stakeholders_variable_influence_interest - raci_matrix - power_interest_grid - communication_plan - - engagement_strategy - - escalation_path quality_checks: - check: raci_completeness @@ -101,7 +100,6 @@ relationships: informs: - prd - risk-register - - communication-plan updated_by: - organizational_changes diff --git a/workflows/activities/01-frame/artifacts/threat-model/meta.yml b/workflows/activities/01-frame/artifacts/threat-model/meta.yml index 70812289..7318ea5a 100644 --- a/workflows/activities/01-frame/artifacts/threat-model/meta.yml +++ b/workflows/activities/01-frame/artifacts/threat-model/meta.yml @@ -18,7 +18,7 @@ output: validation: required_sections: - system_description - - stride_analysis + - stride_threat_analysis - risk_assessment - mitigation_strategies diff --git a/workflows/activities/01-frame/artifacts/threat-model/prompt.md b/workflows/activities/01-frame/artifacts/threat-model/prompt.md index bf004b62..06865f98 100644 --- a/workflows/activities/01-frame/artifacts/threat-model/prompt.md +++ b/workflows/activities/01-frame/artifacts/threat-model/prompt.md @@ -1,5 +1,13 @@ # Threat Modeling Prompt -Document the project threat model with enough detail to drive mitigations. +Enumerate threats by STRIDE category and assign mitigation owners. + +## Traceability chain + +This artifact's place in the security triangle: **threats -> STRIDE + +mitigation owners**. +- See `security-requirements` for the testable controls that mitigate each + threat. +- See `compliance-requirements` for the regulations those mitigations satisfy. ## Reference Anchors @@ -12,18 +20,18 @@ Use these local resource summaries as grounding: ## Focus - Define boundaries, assets, and trust changes first. -- Analyze threats with STRIDE and map them to controls. +- Analyze threats with STRIDE and assign each one a mitigation owner. +- Cross-reference the control(s) in `security-requirements` that mitigate the + threat rather than restating control text. - Keep risk scoring and mitigation ownership explicit. - Treat missing boundaries, unclear assets, or unstated assumptions as findings. - -## Role Boundary - -Threat Model is not Security Requirements or Security Architecture. It explains -what can go wrong and where. Security Requirements state what controls must be -true; Security Architecture places those controls in the design. +- Do not author control acceptance criteria. Cross-reference + `security-requirements`. +- Do not analyze regulatory applicability. Cross-reference + `compliance-requirements`. ## Completion Criteria -- The threat surface is clear. +- The threat surface is clear and threats are categorized by STRIDE. +- Each high-risk threat has a named mitigation owner and a cross-reference to + the mitigating control in `security-requirements`. - Important threats are prioritized. -- Mitigations are concrete. -- High-risk threats have owners and verification hooks. diff --git a/workflows/activities/01-frame/artifacts/user-stories/meta.yml b/workflows/activities/01-frame/artifacts/user-stories/meta.yml index d89c8b43..28d3043d 100644 --- a/workflows/activities/01-frame/artifacts/user-stories/meta.yml +++ b/workflows/activities/01-frame/artifacts/user-stories/meta.yml @@ -38,10 +38,6 @@ validation: description: Every acceptance criterion is independently testable severity: blocking - - check: criteria_have_stable_ids - description: Every acceptance criterion carries a stable US--AC ID so the story test plan and tests can reference it by name - severity: blocking - - check: test_scenarios_concrete description: Test scenarios include concrete values severity: blocking @@ -71,10 +67,6 @@ validation: expected: ">0" message: Must include testable Given/When acceptance criteria - - pattern: "US-[0-9]+-AC[0-9]+" - expected: ">0" - message: Acceptance criteria must carry stable US--AC IDs - - pattern: "[Tt]hen " expected: ">0" message: Must include observable outcomes in acceptance criteria diff --git a/workflows/activities/01-frame/artifacts/user-stories/prompt.md b/workflows/activities/01-frame/artifacts/user-stories/prompt.md index 603e75cc..695aa23a 100644 --- a/workflows/activities/01-frame/artifacts/user-stories/prompt.md +++ b/workflows/activities/01-frame/artifacts/user-stories/prompt.md @@ -140,7 +140,6 @@ committing. - [ ] "So that" names a measurable outcome, not a tautology - [ ] Walkthrough traces a complete path from trigger to outcome - [ ] Every acceptance criterion is independently testable (one Given/When/Then) -- [ ] Every acceptance criterion carries a stable `US--AC` ID - [ ] Test scenarios include concrete values, not placeholders - [ ] Story links to parent feature spec by ID - [ ] Story names the parent feature requirement IDs it exercises diff --git a/workflows/activities/01-frame/artifacts/user-stories/template.md b/workflows/activities/01-frame/artifacts/user-stories/template.md index 67b533cb..8f8b848b 100644 --- a/workflows/activities/01-frame/artifacts/user-stories/template.md +++ b/workflows/activities/01-frame/artifacts/user-stories/template.md @@ -37,12 +37,7 @@ vertical slice — it should cover one complete path from trigger to outcome.] ## Acceptance Criteria -[Each criterion must be testable and carry a **stable AC ID** of the form -`US--AC` (matching this story's number). Use Given/When/Then format — one -precondition, one action, one observable outcome. The ID is stable: it survives -edits so the story test plan and tests can reference a specific criterion by -name. An implementer should be able to write a passing test from each criterion -alone.] +[See the prompt's Acceptance Criteria guidance for the canonical AC-ID rule.] - [ ] **US-XXX-AC1** — Given [specific precondition], when [specific action], then [observable outcome] - [ ] **US-XXX-AC2** — Given [specific precondition], when [specific action], then [observable outcome] diff --git a/workflows/activities/01-frame/artifacts/validation-checklist/meta.yml b/workflows/activities/01-frame/artifacts/validation-checklist/meta.yml index d840a505..94c6c6c4 100644 --- a/workflows/activities/01-frame/artifacts/validation-checklist/meta.yml +++ b/workflows/activities/01-frame/artifacts/validation-checklist/meta.yml @@ -18,12 +18,9 @@ output: validation: required_sections: - - documentation_completeness - - quality_gates - - stakeholder_validation - - cross_reference_validation - - exit_criteria_verification - - sign_off + - go_no_go_gates + - result + - required_follow_up quality_checks: - check: all_artifacts_validated diff --git a/workflows/activities/02-design/README.md b/workflows/activities/02-design/README.md index fda2433a..5a5e6045 100644 --- a/workflows/activities/02-design/README.md +++ b/workflows/activities/02-design/README.md @@ -238,6 +238,34 @@ Technical investigation findings directly inform design artifacts: - **Decision Points**: Clear criteria for when investigation provides sufficient evidence - **Integration Time**: Budget 1-2 days to integrate findings into design artifacts +## Zoom-Stack: What Belongs at Each Level + +Architecture, Solution Design, and Technical Design form a zoom stack over the +same system: Architecture is system-scope, Solution Design is feature-scope, +Technical Design is story-scope. Each level owns the decisions at its scope and +inherits the levels above it. The matrix below is the single source of truth +for which decision type lands in which artifact; the per-artifact prompts +cross-reference this table instead of restating the boundary. + +| Kind of decision | Architecture (system) | Solution Design (feature) | Technical Design (story) | +|---|---|---|---| +| System boundaries, containers, external integrations | Owns | Inherits | Inherits | +| Deployment topology, infrastructure shape, scaling | Owns | Inherits | Inherits | +| System-wide quality attributes and tradeoffs | Owns | Inherits | Inherits | +| Cross-cutting data flow across the whole system | Owns | Inherits | Inherits | +| Feature-level technical approach and option choice | Inherits | Owns | Inherits | +| Domain model and component decomposition for one feature | Inherits | Owns | Inherits | +| Requirement-to-design traceability for a feature | Inherits | Owns | Inherits | +| Cross-component interaction inside one feature | Inherits | Owns | Inherits | +| Acceptance-criterion-to-code mapping for one story | Inherits | Inherits | Owns | +| File, module, and interface changes for one slice | Inherits | Inherits | Owns | +| Story-scoped test approach, rollback, implementation sequence | Inherits | Inherits | Owns | +| One architecturally significant decision with alternatives | ADR | ADR | ADR | +| Exact external interface, schema, or message format | Contract | Contract | Contract | + +If a level needs a decision owned by a higher level, stop and update the +governing artifact first — do not redecide it at the lower level. + ## Artifact Selection Guide Understanding which artifact to use is critical for maintaining clear documentation. This guide helps you choose the right artifact type for your design documentation needs. diff --git a/workflows/activities/02-design/artifacts/architecture/meta.yml b/workflows/activities/02-design/artifacts/architecture/meta.yml index c59512d7..3ed5b6ca 100644 --- a/workflows/activities/02-design/artifacts/architecture/meta.yml +++ b/workflows/activities/02-design/artifacts/architecture/meta.yml @@ -23,8 +23,8 @@ output: validation: required_sections: - - system_context - - container_diagram + - level_1_system_context + - level_2_container_diagram - deployment - data_flow - quality_attributes @@ -99,8 +99,8 @@ prompts: template: file: template.md sections: - - system_context: "Show the system, the actors, and the external systems. Skip if there's only one actor and no externals — that's noise." - - container_diagram: "Required. Each container names its technology, responsibilities, and how it communicates." + - level_1_system_context: "Show the system, the actors, and the external systems. Skip if there's only one actor and no externals — that's noise." + - level_2_container_diagram: "Required. Each container names its technology, responsibilities, and how it communicates." - component_diagram: "Optional. Include only when implementation choices need this level. Most teams ship without it." - deployment: "Required when the system runs anywhere non-trivial. Name regions, instance counts, scaling triggers, backup paths." - data_flow: "One sequence diagram for the most important user journey or operational flow. Pick the one most likely to fail." diff --git a/workflows/activities/02-design/artifacts/architecture/prompt.md b/workflows/activities/02-design/artifacts/architecture/prompt.md index 0f01a3f1..c580f5d3 100644 --- a/workflows/activities/02-design/artifacts/architecture/prompt.md +++ b/workflows/activities/02-design/artifacts/architecture/prompt.md @@ -10,10 +10,8 @@ activity. Its unique job is to describe the durable system shape: boundaries, containers, externally visible integrations, deployment topology, critical data flows, quality attributes, and structural tradeoffs. -Architecture is not a feature solution design, story technical design, ADR, or -runbook. ADRs record individual decisions. Solution and technical designs apply -the architecture to narrower scopes. Operational artifacts inherit deployment -and quality constraints from architecture. +For what belongs at this level versus Solution Design and Technical Design, see +the zoom-stack matrix in `workflows/activities/02-design/README.md`. ## Reference Anchors @@ -35,14 +33,8 @@ Use these local resource summaries as grounding: ## Boundary Test -| If you are writing... | Put it in... | -|---|---| -| Product behavior, priority, or success metrics | PRD or Feature Specification | -| One structural decision with alternatives and consequences | ADR | -| Feature-specific system design | Solution Design | -| Story-level implementation plan | Technical Design | -| Endpoint schemas or message formats | Contract | -| Deployment execution steps, rollback, or release readiness | Deployment Checklist or Runbook | +See the zoom-stack matrix in `workflows/activities/02-design/README.md` for +which decisions belong at the system, feature, and story levels. ## Completion Criteria - The views are understandable at a glance. diff --git a/workflows/activities/02-design/artifacts/data-architecture/example.md b/workflows/activities/02-design/artifacts/data-architecture/example.md index 4e56d015..94c5d8bf 100644 --- a/workflows/activities/02-design/artifacts/data-architecture/example.md +++ b/workflows/activities/02-design/artifacts/data-architecture/example.md @@ -1,8 +1,9 @@ --- ddx: id: example.data-architecture.customer-360 - depends_on: - - example.data-prd.customer-360 + # Previous depends_on: example.data-prd.customer-360 — dropped when + # data-prd collapsed into prd as kind: data variant (ADR-008). No + # equivalent example.prd.customer-360 is yet published. --- # Data Architecture: Customer-360 Analytics diff --git a/workflows/activities/02-design/artifacts/data-architecture/meta.yml b/workflows/activities/02-design/artifacts/data-architecture/meta.yml index 59fd21d4..acd32b8e 100644 --- a/workflows/activities/02-design/artifacts/data-architecture/meta.yml +++ b/workflows/activities/02-design/artifacts/data-architecture/meta.yml @@ -27,9 +27,9 @@ validation: - medallion_topology - data_flow - processing_semantics - - quality_contracts - - governance_and_access - - databricks_platform_design + - pipeline_level_quality_contracts + - governance_and_access_control + - platform_design - decisions_and_tradeoffs quality_checks: @@ -111,9 +111,9 @@ template: - medallion_topology: "Bronze (raw ingestion), Silver (transformed), Gold (consumption) layer strategy" - data_flow: "Diagrams or descriptions of how data moves through layers" - processing_semantics: "Is this streaming, batch, or incremental batch? Why?" - - quality_contracts: "EXPECT rules per layer; what makes data trustworthy?" - - governance_and_access: "Who can access what? Row-level, column-level, catalog policies" - - databricks_platform_design: "Compute tier, storage strategy, catalog/schema organization, SDP expectations" + - pipeline_level_quality_contracts: "EXPECT rules per layer; what makes data trustworthy?" + - governance_and_access_control: "Who can access what? Row-level, column-level, catalog policies" + - platform_design: "Compute tier, storage strategy, catalog/schema organization, SDP expectations" - decisions_and_tradeoffs: "Why these choices? What were the alternatives?" examples: @@ -122,7 +122,9 @@ examples: relationships: informed_by: - - data-prd + # `prd` covers both framings via its `kind` field (ADR-008): `kind: data` + # supplies data-source, consumer, quality, and platform context; default + # `kind: product` supplies general product framing. - prd - data-design @@ -141,8 +143,8 @@ references: file: docs/resources/databricks-lakehouse-architecture.md relationship: "Grounds unified storage, compute, and governance model for data products" - - title: "Databricks Medallion Architecture Pattern" - file: docs/resources/databricks-medallion-architecture.md + - title: "Databricks Lakehouse Medallion Architecture" + file: docs/resources/databricks-lakehouse-medallion-architecture.md relationship: "Grounds Bronze/Silver/Gold transformation semantics and layer isolation" - title: "Databricks Auto Loader and Streaming Tables" diff --git a/workflows/activities/02-design/artifacts/data-architecture/prompt.md b/workflows/activities/02-design/artifacts/data-architecture/prompt.md index 540db2c9..2e1f5436 100644 --- a/workflows/activities/02-design/artifacts/data-architecture/prompt.md +++ b/workflows/activities/02-design/artifacts/data-architecture/prompt.md @@ -12,7 +12,7 @@ transformation patterns, governance boundaries, quality gates, and critical performance or cost tradeoffs. Data Architecture is not a data model (captured in Data Design), implementation plan, -or ADR. It is the bridge between Data PRD (requirements) and implementation: "given +or ADR. It is the bridge between PRD (kind: data) (requirements) and implementation: "given these requirements, here is how the pipeline is structured." ## Reference Anchors @@ -49,17 +49,10 @@ Data Architecture describes pipeline topology and data flow, not the detailed data model (Data Design), not implementation sequences (Implementation Plan), and not individual quality checks (Data Quality Expectations). -**Databricks Platform Substitution:** If you are adopting this on another data -platform, substitute as follows: - -| Databricks Concept | Snowflake Equivalent | BigQuery Equivalent | On-Prem / Other | -|---|---|---|---| -| Medallion layers (Bronze/Silver/Gold) | Same pattern applies universally | Same pattern applies universally | Same pattern applies universally | -| Auto Loader | Snowpipe or native connectors | Dataflow, BigQuery Connector Hub | Apache NiFi, Kafka connectors | -| Streaming Tables with `EXPECT` clauses | Stream-triggered materialized views + native checks | Dataflow with Beam assertions | Apache Flink with custom state management | -| Databricks Jobs for orchestration | Snowflake Tasks | Cloud Composer (Airflow) or Cloud Workflows | Apache Airflow, Dagster, dbt Cloud | -| SDP `EXPECT ... ON VIOLATION ...` | Data Quality checks + Task error handling | BigQuery Data Quality API + Cloud Workflows | dbt tests, Great Expectations, custom assertions | -| Delta Lake format | Iceberg or proprietary formats | Native BigQuery tables | Apache Parquet, Iceberg, Hudi | +**Non-Databricks platforms:** see +`docs/resources/databricks-platform-substitution.md` for the equivalent +terms on Snowflake, BigQuery, and on-prem stacks. The artifact shape and +prompt stay the same. ## Completion Criteria @@ -71,4 +64,4 @@ platform, substitute as follows: - Performance/cost tradeoffs are visible (partitioning, clustering, retention, refresh strategy). - Deployment topology is concrete (number of clusters, auto-scaling, failover). -- Major decisions link to Data PRD requirements or include inline rationale. +- Major decisions link to PRD (kind: data) requirements or include inline rationale. diff --git a/workflows/activities/02-design/artifacts/data-architecture/template.md b/workflows/activities/02-design/artifacts/data-architecture/template.md index 8a1123a1..7e4dd89f 100644 --- a/workflows/activities/02-design/artifacts/data-architecture/template.md +++ b/workflows/activities/02-design/artifacts/data-architecture/template.md @@ -5,129 +5,114 @@ ddx: # Data Architecture +Platform- and pipeline-level shape of the data product: medallion topology, +processing-framework choices, governance model, and pipeline-level quality +contracts. Entity-level modelling (logical schema, access patterns, +constraints, migration) lives in [[data-design]]. + ## Overview -[Describe the data product being architected, the business problem it solves, and the system context. Name the key data flows and Databricks platform fit. Reference the [[data-prd]] for the business requirements and success metrics this architecture must satisfy.] +[Describe the data product being architected, the business problem it solves, +and the system context. Name the key data flows and platform fit. Reference +[[prd]] (kind: data) for the requirements and success metrics this architecture must +satisfy.] ### Scope -[What data flows and systems are covered by this architecture? What is deliberately outside the boundary? Which requirements from [[data-prd]] drive the design decisions?] +[What data flows and systems are covered. What is deliberately out of bounds. +Which requirements from [[prd]] (kind: data) drive the design decisions.] ### System Context | External System | Role | Protocol | Data Volume | |-----------------|------|----------|------------| -| [e.g., Salesforce] | [Source system for customer data] | [REST API, batch export] | [M records/day] | -| [e.g., Data Warehouse] | [Consumer of aggregated data] | [Delta API, SQL] | [K rows/hour] | +| [Source system] | [Role in the pipeline] | [API, batch export, CDC] | [Order-of-magnitude per period] | +| [Consumer system] | [How it consumes Gold] | [Delta share, SQL, API] | [Query volume] | ```mermaid graph TB - A["Salesforce
(Source)"] -->|REST API
hourly| B["Databricks
Data Platform"] - C["Stripe
(Source)"] -->|S3 batch export| B - B -->|Consumption Layer| D["BI Tool
(Looker/Tableau)"] - B -->|ML Training| E["ML Platform"] + A["Source A"] -->|ingest| B["Data Platform"] + C["Source B"] -->|ingest| B + B -->|consumption layer| D["BI / Reporting"] + B -->|feature store| E["ML Platform"] ``` ## Medallion Topology -### Architecture Pattern +### Layer Strategy -[Describe the medallion layer strategy: Bronze (raw), Silver (validated), Gold (consumption). For each layer, explain the transformation scope, quality gates, and consumer responsibilities.] +[State the medallion strategy: Bronze (raw), Silver (validated), Gold +(consumption). For each layer, name the transformation scope, quality gates, +and consumer responsibilities. Justify the choice against [[prd]] (kind: data) +freshness and quality requirements.] ### Bronze Layer (Raw Ingestion) -**Purpose**: Land source data in its native form without transformation. - -**Source Integration**: -- Source System: [e.g., Salesforce] -- Integration Pattern: [Auto Loader, Streaming Tables, scheduled batch import] -- Schema Validation: [Applied at ingestion, reject if schema mismatch] -- Retention: [e.g., 7 days rolling window for cost efficiency] +- **Purpose**: Land source data in its native form without transformation. +- **Source integration pattern**: [Auto Loader, Streaming Tables, scheduled + batch import, CDC] +- **Schema handling**: [Strict / inferred / evolution policy] +- **Retention policy**: [Rationale tied to cost and replay needs] -**Responsibilities**: -- Ingest all records from source -- Preserve source schema exactly (no column renames or type coercion) -- Tag records with `_ingest_timestamp` and source system identifier -- Detect and quarantine records that fail schema validation +Responsibilities: +- Ingest all records from source. +- Preserve source schema exactly (no renames or coercion). +- Tag records with ingest timestamp and source-system identifier. +- Quarantine records that fail schema validation. -**Quality Gates**: -- All records have `_ingest_timestamp` and source system id -- No truncation of source columns -- Fail fast if source unavailable for >4 hours +Quality gates: ingest-metadata presence, no column truncation, source +availability watchdog. ### Silver Layer (Validated and Transformed) -**Purpose**: Cleansed, deduplicated, and business-logic-ready data. - -**Transformations**: -- Deduplication: [Method: e.g., "keep latest by timestamp", "PK uniqueness constraint"] -- Data type coercion: [e.g., "convert dates to ISO-8601, currency to numeric"] -- Null handling: [e.g., "fill defaults, or fail if critical column null"] -- Referential integrity: [e.g., "customer_id must exist in customer dimension"] - -**Join Strategy**: -| Join | Left | Right | Type | Cardinality | Latency Impact | -|------|------|-------|------|-------------|---| -| [customer enrichment] | `customer_events` | `customer_master` | [Left Outer] | [1:1] | [Add 100ms] | +- **Purpose**: Cleansed, deduplicated, business-logic-ready data. +- **Deduplication strategy**: [Key + ordering rule] +- **Type coercion / null policy**: [Defaults vs reject] +- **Referential integrity**: [Which FK relationships are enforced and how] -**Retention**: [e.g., 90 days] +Join strategy (pipeline-level — entity-level joins live in [[data-design]]): -**Responsibilities**: -- Enforce UNIQUE and NOT NULL constraints per data quality expectations -- Materialize aggregations needed for Gold layer -- Document row counts and latency metrics per transformation +| Join | Source Layers | Type | Cardinality | Latency Impact | +|------|---------------|------|-------------|----------------| +| [Logical join name] | [Left / Right] | [Inner / Outer] | [1:1 / 1:N] | [Qualitative] | -**Quality Gates**: -- Zero duplicates (PK uniqueness) -- ≥99% NOT NULL on critical columns -- Row count reconciliation with source ±0.1% +Quality gates: PK uniqueness, NOT NULL on critical columns, row-count +reconciliation with Bronze within tolerance. ### Gold Layer (Consumption) -**Purpose**: Business-ready tables optimized for consumer queries. +- **Purpose**: Business-ready tables optimised for consumer queries. +- **Optimisation strategy**: [Partitioning, clustering / z-order, materialised + views — at the pipeline level, not column-level] +- **Retention policy**: [Compliance and analytics horizon] -**Consumption Tables**: -| Table | Use Case | Consumers | Freshness | Aggregation | -|-------|----------|-----------|-----------|------------| -| `customer_360` | Single customer view, 360-degree profile | Analytics, ML | Hourly | None (deduped Silver) | -| `daily_sales_summary` | Daily revenue by product | Finance, BI | Daily | SUM(amount) by product_date | +Consumption tables (entity definitions live in [[data-design]]): -**Optimization Strategy**: -- Partitioning: [e.g., "by date for date-range queries"] -- Z-order: [e.g., "on customer_id, product_id for filter selectivity"] -- Caching: [Materialized views for slow queries] +| Table | Use Case | Consumers | Freshness Target | +|-------|----------|-----------|------------------| +| [Gold table name] | [Use case from PRD (kind: data)] | [Persona] | [Target tied to SLA] | -**Retention**: [e.g., 2 years for compliance and analytics] - -**Responsibilities**: -- Keep Gold current with Silver ingestion cadence -- Publish schema and SLA via UC catalog comments -- Monitor query performance; alert if p95 latency > SLA - -**Quality Gates**: -- Sums reconcile between Silver aggregates and Gold tables ±0.01% -- All customer IDs in Gold exist in Silver Silver -- Latency ≤SLA (e.g., ≤5 seconds for 90th percentile query) +Quality gates: aggregate reconciliation with Silver, referential integrity +across Gold, latency within consumer SLA. ## Data Flow -[Describe how data moves through the medallion layers. Clarify ingestion frequency, transformation latency, and refresh strategy.] - -### Ingestion Flow +[Describe how data moves through the medallion layers. Clarify ingestion +frequency, transformation latency, and refresh strategy.] ```mermaid graph LR - A["Salesforce API"] -->|Auto Loader
every 15 min| B["Bronze
raw_customers"] - B -->|PySpark Job
dedup + validate| C["Silver
customers_validated"] - C -->|SQL notebook
daily 9am UTC| D["Gold
customer_360"] - D -->|Published to UC| E["Consumers
BI, ML"] + A["Source"] -->|ingest pattern| B["Bronze"] + B -->|transform job| C["Silver"] + C -->|aggregate job| D["Gold"] + D -->|published| E["Consumers"] ``` ### Incremental vs Full Refresh -- **Bronze**: [Incremental via change data capture, or full reload?] -- **Silver**: [Incremental updates on specific key columns, or full recalc?] -- **Gold**: [Append-only fact tables, or snapshot updates?] +- **Bronze**: [CDC / append / full reload — rationale] +- **Silver**: [Incremental keys / full recalc — rationale] +- **Gold**: [Append-only / snapshot / merge — rationale] ## Processing Semantics @@ -135,206 +120,188 @@ graph LR | Layer | Strategy | Rationale | SLA Implication | |-------|----------|-----------|-----------------| -| Bronze | [Streaming / Batch / Incremental Batch] | [Why this choice?] | [Freshness achieved] | -| Silver | [Streaming / Batch / Incremental Batch] | [Why this choice?] | [Freshness achieved] | -| Gold | [Streaming / Batch / Incremental Batch] | [Why this choice?] | [Freshness achieved] | +| Bronze | [Streaming / Batch / Incremental] | [Why] | [Freshness achieved] | +| Silver | [Streaming / Batch / Incremental] | [Why] | [Freshness achieved] | +| Gold | [Streaming / Batch / Incremental] | [Why] | [Freshness achieved] | ### Processing Framework -- **Framework**: [Databricks SQL, PySpark, dbt on Databricks, Streaming Tables] -- **Orchestration**: [Databricks Workflows, Airflow, dbt Cloud] -- **Failure Handling**: [Retry policy, dead-letter queue, manual intervention?] +- **Framework**: [Databricks SQL, PySpark, dbt, Streaming Tables, Flink, …] +- **Orchestration**: [Workflows, Airflow, dbt Cloud, Dagster, …] +- **Failure handling**: [Retry policy, dead-letter queue, manual intervention] +- **Idempotence / exactly-once posture**: [Per layer] +- **Schema evolution policy**: [Auto-add / manual approval / strict] ### Latency and Throughput Targets -| Stage | Target Latency | Target Throughput | Constraint | -|-------|-----------------|-------------------|-----------| -| Salesforce → Bronze | ≤15 minutes | 1M records/day | API rate limit: 100 req/min | -| Bronze → Silver | ≤30 minutes | 1M records/day | PySpark cluster size | -| Silver → Gold | ≤2 hours | 100K aggregates/day | SQL query complexity | - -## Quality Contracts +| Stage | Latency Target | Throughput Target | Binding Constraint | +|-------|----------------|-------------------|--------------------| +| Source → Bronze | [From PRD (kind: data) SLA] | [Order of magnitude] | [Rate limit, API quota] | +| Bronze → Silver | [From PRD (kind: data) SLA] | [Order of magnitude] | [Compute / dedup cost] | +| Silver → Gold | [From PRD (kind: data) SLA] | [Order of magnitude] | [Query complexity] | -[Define expectations as testable EXPECT clauses per Databricks SDP. Each expectation binds the architecture: data violating it is rejected before reaching consumers.] +## Pipeline-Level Quality Contracts -### Bronze Layer Expectations +[Express the contracts the *pipeline* enforces at each layer boundary. +Column-level field rules belong in [[data-quality-expectations]]; this +section names which contracts the architecture commits to enforce and +where.] -```sql --- Raw data must match source schema exactly -EXPECT TABLE raw_customers ( - customer_id NOT NULL, - email NOT NULL, - created_at TIMESTAMP NOT NULL, - _ingest_timestamp TIMESTAMP NOT NULL GENERATED ALWAYS AS (CURRENT_TIMESTAMP()) -); -``` - -- **Expectation**: All records ingested within 15 min of source commit -- **Violation**: Alert; do not advance to Silver -- **SLA**: Detect within 5 minutes - -### Silver Layer Expectations - -```sql --- Customers must be unique per customer_id, deduplicated by latest timestamp -EXPECT TABLE customers_validated ( - customer_id NOT NULL, - PRIMARY KEY (customer_id), - UNIQUE (customer_id) -); +### Bronze → Silver --- Customer email addresses must be normalized and not null -EXPECT TABLE customers_validated ( - email NOT NULL LIKE '%@%.%' -); -``` +- **Schema contract**: [What Silver requires of Bronze] +- **Volume contract**: [Acceptable row-count delta] +- **Freshness contract**: [Max ingest lag before Silver is held] +- **Violation handling**: [Alert / hold / quarantine] -- **Expectation**: ≥99% NOT NULL on `email` and `phone` -- **Expectation**: Zero duplicate customers (PK uniqueness) -- **Violation**: Quarantine; manual review before advancing to Gold - -### Gold Layer Expectations - -```sql --- Customer 360 must be current within freshness SLA (1 hour) -EXPECT TABLE customer_360 ( - MAX(_modified_at) >= CURRENT_TIMESTAMP() - INTERVAL 1 HOUR -); - --- Revenue aggregates must reconcile with Silver within 0.01% -EXPECT TABLE daily_sales_summary - CHECK ( - SELECT SUM(amount) FROM daily_sales_summary - IS WITHIN 0.01% OF - SELECT SUM(amount) FROM customers_validated - ); -``` +### Silver → Gold -- **Expectation**: Customer 360 is no older than 1 hour -- **Expectation**: Daily sales sums reconcile with Silver ±$0.01 per order -- **Violation**: Reject; roll back to prior snapshot; alert on-call +- **Uniqueness contract**: [Which keys are unique at Gold] +- **Referential contract**: [Which FK relationships are guaranteed] +- **Aggregate-reconciliation contract**: [Sums and counts must agree within + tolerance] +- **Violation handling**: [Reject / rollback / alert] -### Cross-Layer Data Contracts +### Cross-Layer Contracts | Contract | Assertion | If Violated | -|----------|-----------|------------| -| [Bronze → Silver row count] | [Silver row count ≤ Bronze + 10% for dedup] | [Alert + manual audit] | -| [Silver → Gold cardinality] | [Gold unique customers = Silver unique customers] | [Reject until reconciled] | -| [Foreign key integrity] | [All orders.customer_id exist in customer_360] | [Quarantine order; alert] | +|----------|-----------|-------------| +| [Row count Bronze → Silver] | [Within tolerance] | [Alert + manual audit] | +| [Cardinality Silver → Gold] | [Stable across refresh] | [Reject until reconciled] | +| [FK integrity across Gold] | [No orphans] | [Quarantine + alert] | + +Detailed `EXPECT` clauses, field-level constraints, and freshness predicates +live in [[data-quality-expectations]]. ## Governance and Access Control -### Identity and Access Management +### Identity and Access Model -| Role | Catalog | Schema | Table | Permissions | -|------|---------|--------|-------|------------| -| Data Engineer | `prod` | `customer_360` | All | READ, MODIFY, EXECUTE | -| Analytics Lead | `prod` | `customer_360` | `customer_360`, `daily_sales_summary` (Gold only) | SELECT | -| Finance Team | `prod` | `customer_360` | `daily_sales_summary` | SELECT (date ≥ 2 years ago) | +| Role | Catalog Scope | Layer Access | Permissions | +|------|---------------|--------------|-------------| +| [Role from PRD (kind: data) consumers] | [Catalog / schema] | [Bronze / Silver / Gold] | [SELECT / MODIFY / EXECUTE] | ### Data Classification and Retention -| Table | Classification | Sensitive Columns | Retention | Masked For | -|-------|------------------|------------------|-----------|-----------| -| `raw_customers` | Internal | `ssn`, `credit_card` | 7 days | [Not masked; only admins see] | -| `customers_validated` | Internal | `email`, `phone` | 90 days | [Non-PII audiences mask these] | -| `customer_360` | Internal | `email` | 2 years | [Mask for BI tool] | +| Layer | Classification | Sensitive Categories | Retention Policy | Masking Policy | +|-------|----------------|----------------------|------------------|----------------| +| Bronze | [Class] | [Categories — not specific columns; those live in data-design] | [Policy tied to compliance] | [Who sees raw] | +| Silver | [Class] | [Categories] | [Policy] | [Who sees masked vs raw] | +| Gold | [Class] | [Categories] | [Policy] | [Default masking for BI] | ### Fine-Grained Access Control -- **Row-Level Security**: [Do analytics users see only their region's data? Document the policy.] -- **Column-Level Security**: [Which sensitive columns are masked for which roles?] -- **Dynamic Views**: [Use UC masking functions to redact PII per caller role?] +- **Row-level security**: [Tenant / region predicate — policy, not the + predicate code, which lives in [[data-design]]] +- **Column-level security**: [Which classifications are masked for which + roles] +- **Dynamic views**: [Masking-function strategy] -## Databricks Platform Design +## Platform Design -### Catalog Organization +### Catalog Organisation ``` -prod (Catalog) -├── customer_360 (Schema) -│ ├── raw_customers (Bronze table) -│ ├── customers_validated (Silver table) -│ ├── customer_360 (Gold table) -│ └── daily_sales_summary (Gold aggregate) -├── metadata (Schema) -│ ├── pipeline_runs (audit log) -│ └── quality_metrics (expectations results) +[catalog] +├── [schema] +│ ├── [bronze table family] +│ ├── [silver table family] +│ └── [gold table family] +├── metadata +│ ├── pipeline_runs +│ └── quality_metrics ``` ### Compute Strategy -| Workload | Compute Tier | Cluster Size | DBU Budget | Rationale | -|----------|--------------|--------------|------------|-----------| -| Bronze ingestion (streaming) | All-purpose | 4 workers (i3.xlarge) | 8 DBU/hour | Continuous streaming | -| Silver transformation (batch) | Jobs | 8 workers (i3.2xlarge) | 16 DBU/run | Scheduled nightly | -| Gold aggregation (query) | Serverless SQL | N/A | ~5 DBU/query | Ad-hoc BI queries | +| Workload | Compute Tier | Sizing Approach | Rationale | +|----------|--------------|-----------------|-----------| +| Bronze ingestion | [Tier] | [Auto-scale bounds / fixed] | [Continuous vs scheduled] | +| Silver transformation | [Tier] | [Sizing approach] | [Batch vs streaming] | +| Gold consumption | [Tier] | [Sizing approach] | [Query pattern] | + +Cost-shaping levers (qualitative — concrete numbers belong in operational +runbooks, not the architecture): -**Cost Optimization**: -- Spot instances for non-critical workloads: 30% savings -- Auto-terminate idle clusters: reduce waste -- Partition pruning and z-order for faster queries: reduce scans +- Spot / preemptible instances for retryable workloads. +- Auto-termination of idle clusters. +- Partition pruning and clustering for scan reduction. +- Materialised vs on-demand aggregates. ### Storage Strategy -| Layer | Format | Compression | Optimization | -|-------|--------|-------------|--------------| -| Bronze | Delta | Snappy | Partitioned by date (rolling 7-day window) | -| Silver | Delta | Snappy | Z-ordered by customer_id, product_id | -| Gold | Delta | Snappy | Partitioned by date; cached for BI queries | +| Layer | Format | Partitioning | Clustering / Optimisation | +|-------|--------|--------------|---------------------------| +| Bronze | [Delta / Iceberg / …] | [By date / source] | [Compaction policy] | +| Silver | [Format] | [By key / date] | [Z-order / cluster keys] | +| Gold | [Format] | [By query predicate] | [Materialised views / cache] | -**Storage Sizing**: ~500 GB Bronze, ~50 GB Silver, ~5 GB Gold (monthly) → ~$25/month at $0.023/GB/month +### Platform Features in Use -### Databricks Features in Use +| Feature | Use Case | Configuration Note | +|---------|----------|--------------------| +| [Auto Loader / equivalent] | [Bronze ingestion] | [Trigger mode, schema mode] | +| [Streaming Tables / equivalent] | [Bronze → Silver] | [Trigger / latency target] | +| [Pipeline orchestrator] | [End-to-end refresh] | [Schedule / dependency] | +| [Governance catalog] | [Access + lineage] | [Cross-team sharing posture] | -| Feature | Use Case | Configuration | -|---------|----------|---------------| -| Auto Loader | Salesforce → Bronze incremental ingestion | Cloud file location: S3, Schema inference enabled | -| Streaming Tables | Bronze → Silver continuous transformation | Trigger: arrival, 30-second max latency | -| Delta Live Tables | End-to-end medallion pipeline orchestration | Refresh: hourly for Bronze/Silver, daily for Gold | -| Unity Catalog | Governance, lineage, fine-grained access | Enable open sharing across teams | +For non-Databricks platforms, see +[`docs/resources/databricks-platform-substitution.md`](../../../../../docs/resources/databricks-platform-substitution.md) +for the platform-equivalent terms. ## Decisions and Tradeoffs ### Key Architecture Decisions | Decision | Choice | Rationale | Alternative Considered | Consequence | -|----------|--------|-----------|------------------------|------------| -| [Medallion layers] | [Bronze/Silver/Gold] | [Standard Databricks pattern for quality gates] | [Flat schema, 2-layer] | [Slower queries if using Bronze directly; quality risk] | -| [Streaming vs Batch for Silver] | [Batch nightly] | [Simplicity, cost] | [Streaming] | [1-2 hour stale window; acceptable for reporting] | -| [Compute tier for Gold queries] | [Serverless SQL] | [Cost & elasticity; no cluster mgmt] | [All-purpose cluster] | [Higher per-query cost; simpler ops] | +|----------|--------|-----------|------------------------|-------------| +| [Medallion layering] | [Choice] | [Why] | [Alternative] | [Tradeoff] | +| [Streaming vs batch per layer] | [Choice] | [Why] | [Alternative] | [Tradeoff] | +| [Compute tier per workload] | [Choice] | [Why] | [Alternative] | [Tradeoff] | ### Performance vs Cost Tradeoffs -- **Real-time ingestion** (streaming Bronze): Freshness ≤5 min, but 40% higher DBU cost -- **Materialized Gold aggregates**: Faster queries (100ms), but 20% higher storage cost -- **Spot instances for Silver jobs**: 30% cheaper, but risk of interruption (retry handles it) +- [Real-time vs near-real-time ingestion — freshness gain vs sustained + compute cost] +- [Materialised vs on-demand Gold aggregates — query latency vs storage] +- [Spot vs on-demand compute — cost savings vs interruption risk] ### Known Risks and Mitigations | Risk | Mitigation | -|------|-----------| -| [Salesforce API rate limit causes ingestion backlog] | [Implement exponential backoff; buffer in queue; alert if >1 hour backlog] | -| [PII in Bronze accessible to all engineers] | [Mask sensitive columns in published views; audit access logs] | -| [Schema drift in source breaks ingestion] | [Schema registry; auto-detect new columns; manual approval before Silver] | +|------|------------| +| [Source rate limit causes backlog] | [Backoff + queue buffering + lag alert] | +| [PII exposure in Bronze] | [Masked views + audit logs] | +| [Schema drift from source] | [Schema registry + manual approval gate] | --- ## Review Checklist -Use this checklist during review to validate that the data architecture is complete and ready for implementation: - -- [ ] **Scope** clearly states which data flows are included and which are out of bounds -- [ ] **Medallion Topology** defines Bronze, Silver, Gold layer purposes and transformation rules -- [ ] **Data Flow diagrams** (mermaid) show how data moves through layers and to consumers -- [ ] **Processing Semantics** explicitly state streaming vs batch for each layer with latency targets -- [ ] **Quality Contracts** are written as executable EXPECT clauses (SDP, dbt, SQL constraints) -- [ ] **Failure Handling** specifies what happens when an expectation fails (alert, reject, quarantine) -- [ ] **Access Control** model covers identity, row-level, column-level, and sensitive data masking -- [ ] **Databricks Platform Design** names catalog, schema, compute tier, storage strategy, and DBU budget -- [ ] **Decisions and Tradeoffs** document key choices with rationale and alternatives considered -- [ ] **Cross-layer data contracts** are defined (sums reconcile, cardinality stable, no orphans) -- [ ] **SLA per layer** is documented (freshness, latency, availability targets) -- [ ] No `[TBD]`, `[TODO]`, or `[NEEDS CLARIFICATION]` markers remain -- [ ] Architecture aligns with requirements in [[data-prd]] and can satisfy success metrics -- [ ] References link to [[data-quality-expectations]] for detailed EXPECT clauses per layer +- [ ] **Scope** clearly states which data flows are in / out of bounds. +- [ ] **Medallion topology** names Bronze / Silver / Gold purposes and + transformation rules. +- [ ] **Data flow diagrams** show how data moves through layers and to + consumers. +- [ ] **Processing semantics** explicitly state streaming vs batch per layer + with latency targets tied to [[prd]] (kind: data). +- [ ] **Pipeline-level quality contracts** name which contracts each layer + boundary enforces; detailed `EXPECT` clauses are deferred to + [[data-quality-expectations]]. +- [ ] **Failure handling** specifies what happens when a contract fails + (alert, reject, quarantine, rollback). +- [ ] **Access control** model covers identity, row-level, column-level, + and sensitive-data masking at the policy level. +- [ ] **Platform design** names catalog organisation, compute tiering, and + storage strategy without committing to hardcoded cost numbers. +- [ ] **Decisions and tradeoffs** document key choices with rationale and + alternatives considered. +- [ ] **Cross-layer contracts** are defined (reconciliation, cardinality, + no orphans). +- [ ] **SLA per layer** is documented (freshness, latency, availability) + and traces to [[prd]] (kind: data). +- [ ] No `[TBD]`, `[TODO]`, or `[NEEDS CLARIFICATION]` markers remain. +- [ ] Entity-level details (logical schema, indexes, migrations, store + selection) are deferred to [[data-design]]. +- [ ] For non-Databricks platforms, terms map via + `docs/resources/databricks-platform-substitution.md`. diff --git a/workflows/activities/02-design/artifacts/data-design/template.md b/workflows/activities/02-design/artifacts/data-design/template.md index 6da750a8..4b60b138 100644 --- a/workflows/activities/02-design/artifacts/data-design/template.md +++ b/workflows/activities/02-design/artifacts/data-design/template.md @@ -5,10 +5,17 @@ ddx: # Data Design +Entity-level data model for the feature or subsystem: logical entities, +stores, relationships, access patterns, integrity/security constraints, and +migration strategy. Platform-level concerns (medallion topology, processing +framework, governance model, pipeline-level quality contracts) live in +[[data-architecture]]. + ## Data Summary - Scope: [What feature, subsystem, or workflow this data design supports] -- Storage systems: [Database, queue, cache, object store] +- Storage systems: [Database, queue, cache, object store — names only; the + platform-level rationale lives in [[data-architecture]]] - Main concerns: [Consistency, scale, retention, privacy, migration] ## Entities and Stores @@ -31,6 +38,9 @@ ddx: ## Validation and Security +Field-level rules. Pipeline-level masking and access policy live in +[[data-architecture]] (Governance and Access Control). + | Field or Data Type | Rules / Classification | Protection or Error Handling | |--------------------|------------------------|------------------------------| | [Field] | [Constraints or classification] | [Masking, encryption, validation, retention] | @@ -40,3 +50,11 @@ ddx: - Tooling: [Migration framework] - Approach: [Schema rollout and rollback strategy] - Backfill or cleanup: [If needed] + +## Cross-References + +- [[data-architecture]] — platform/pipeline shape, medallion topology, + processing framework, governance model, and pipeline-level quality + contracts. +- [[data-quality-expectations]] — executable field-level and freshness + contracts that this model must satisfy. diff --git a/workflows/activities/02-design/artifacts/proof-of-concept/meta.yml b/workflows/activities/02-design/artifacts/proof-of-concept/meta.yml index 95eae16b..e0fffab8 100644 --- a/workflows/activities/02-design/artifacts/proof-of-concept/meta.yml +++ b/workflows/activities/02-design/artifacts/proof-of-concept/meta.yml @@ -21,12 +21,10 @@ output: validation: required_sections: - objective - - scope - approach - implementation - - testing - - findings - - conclusions + - results + - analysis - recommendations - artifacts diff --git a/workflows/activities/02-design/artifacts/proof-of-concept/template.md b/workflows/activities/02-design/artifacts/proof-of-concept/template.md index e37362aa..9738a0d8 100644 --- a/workflows/activities/02-design/artifacts/proof-of-concept/template.md +++ b/workflows/activities/02-design/artifacts/proof-of-concept/template.md @@ -74,3 +74,10 @@ ddx: ### Follow-up - [ ] Design updates, additional PoCs, ADRs to create + +## Artifacts + +Preserved under `proofs-of-concept/{{poc_id}}/`: +- **Code**: [Working implementation files] +- **Data**: [Test data and fixtures] +- **Evidence**: [Logs, screenshots, benchmark output] diff --git a/workflows/activities/02-design/artifacts/solution-design/prompt.md b/workflows/activities/02-design/artifacts/solution-design/prompt.md index 599e10ba..dbafeb79 100644 --- a/workflows/activities/02-design/artifacts/solution-design/prompt.md +++ b/workflows/activities/02-design/artifacts/solution-design/prompt.md @@ -10,8 +10,9 @@ model, component decomposition, interface usage, and requirement-to-design traceability. It applies Architecture, ADRs, Contracts, and Concerns to one feature or -cross-component capability. It does not redefine the system architecture. It -does not plan a single story's code changes; that belongs in Technical Design. +cross-component capability. For what belongs at this level versus Architecture +and Technical Design, see the zoom-stack matrix in +`workflows/activities/02-design/README.md`. ## Reference Anchors @@ -27,22 +28,15 @@ Use these local resource summaries as grounding: - Show the main options and why the chosen one wins. - Keep the domain model, decomposition, and tradeoffs concise. - Cover cross-component system behavior and feature-level structure. -- Do not collapse into story-level implementation details; those belong in a - technical design (`TD-XXX-*`). - Preserve only the decisions needed by build and test. -- If the feature requires changing Architecture or an accepted ADR, stop and - update that governing artifact first. +- Stay within the feature scope defined in the zoom-stack matrix + (`workflows/activities/02-design/README.md`); update the governing artifact + first if the feature forces a change at a higher level. ## Boundary Test -| If you are writing... | Put it in... | -|---|---| -| Product/feature behavior and acceptance criteria | Feature Specification | -| System-wide structure, deployment, and quality attributes | Architecture | -| One architecture-significant decision | ADR | -| Exact API/file/event/CLI surface | Contract | -| Feature-level approach and decomposition | Solution Design | -| Story-level code plan and file/module changes | Technical Design | +See the zoom-stack matrix in `workflows/activities/02-design/README.md` for +which decisions belong at the system, feature, and story levels. ## Completion Criteria - Requirements are mapped. diff --git a/workflows/activities/02-design/artifacts/tech-spike/meta.yml b/workflows/activities/02-design/artifacts/tech-spike/meta.yml index 51159336..adfa4984 100644 --- a/workflows/activities/02-design/artifacts/tech-spike/meta.yml +++ b/workflows/activities/02-design/artifacts/tech-spike/meta.yml @@ -23,6 +23,7 @@ validation: - hypothesis - approach - findings + - analysis - conclusions - recommendations - artifacts diff --git a/workflows/activities/02-design/artifacts/technical-design/example.md b/workflows/activities/02-design/artifacts/technical-design/example.md index 0c88c541..bfd42cfe 100644 --- a/workflows/activities/02-design/artifacts/technical-design/example.md +++ b/workflows/activities/02-design/artifacts/technical-design/example.md @@ -27,18 +27,6 @@ ddx: - Does not implement column mapping, row validation, import confirmation, or match generation. -## Acceptance Criteria - -1. **Given** Maya is viewing Acme Dental, **When** she uploads one valid bank - CSV and one valid invoice CSV, **Then** DepositMatch creates one draft import - session for Acme Dental and opens mapping review. -2. **Given** Maya is viewing Acme Dental, **When** she uploads a PDF instead of - a CSV for either required file, **Then** DepositMatch rejects the file before - parsing and keeps the import session in draft. -3. **Given** Maya has uploaded both required CSV files, **When** the files are - accepted, **Then** the import session records the client, file names, upload - time, and source type for each file. - ## Technical Approach **Strategy**: Implement the API-001 upload contract in the Fastify API and add a @@ -168,16 +156,18 @@ CREATE TABLE import_files ( ## Testing -- [ ] **Unit**: `importUploadService` rejects missing files, non-CSV files, and - inaccessible clients. -- [ ] **Integration**: API route returns API-001 success shape and stores draft - session/file metadata in one transaction. -- [ ] **API**: Contract tests for 201, 400 `missing-import-file`, 415 - `unsupported-import-file-type`, and 503 `import-storage-unavailable`. +Each governing-story AC-ID is realized below (ADR-009 — AC text lives in [[US-001]], not here): + +- [ ] **Unit** (US-001-AC1, US-001-AC2): `importUploadService` rejects missing + files, non-CSV files, and inaccessible clients. +- [ ] **Integration** (US-001-AC1, US-001-AC3): API route returns API-001 + success shape and stores draft session/file metadata in one transaction. +- [ ] **API** (US-001-AC2): Contract tests for 201, 400 `missing-import-file`, + 415 `unsupported-import-file-type`, and 503 `import-storage-unavailable`. - [ ] **Security**: Verify raw CSV row values are absent from logs for failed and successful uploads. -- [ ] **UI**: Upload component routes to `next.href` after successful upload and - renders problem-details errors. +- [ ] **UI** (US-001-AC1): Upload component routes to `next.href` after + successful upload and renders problem-details errors. ## Migration & Rollback diff --git a/workflows/activities/02-design/artifacts/technical-design/meta.yml b/workflows/activities/02-design/artifacts/technical-design/meta.yml index 8528a4ae..a6529f7b 100644 --- a/workflows/activities/02-design/artifacts/technical-design/meta.yml +++ b/workflows/activities/02-design/artifacts/technical-design/meta.yml @@ -24,16 +24,13 @@ output: validation: required_sections: - - story_reference - - acceptance_criteria_review - technical_approach - component_changes - - testing_approach - - definition_of_done + - testing quality_checks: - - check: acceptance_criteria_mapped - description: Every acceptance criterion must map to technical implementation + - check: acceptance_criteria_realized + description: Each governing-story AC-ID (US-{n}-AC{m}) is realized by the technical changes; AC text is not restated here (ADR-009 — AC ownership lives in user-stories). severity: blocking - check: api_contracts_defined @@ -61,9 +58,9 @@ validation: expected: ">=1" message: Must reference the solution design - - pattern: "acceptance criteri" + - pattern: "US-\\d{3}-AC\\d+" expected: ">=1" - message: Must review acceptance criteria + message: Must reference at least one governing story AC-ID (US-{n}-AC{m}); do not restate AC text (ADR-009) cross_references: must_reference: @@ -103,13 +100,11 @@ prompts: template: file: template.md sections: - - story_reference: "Link to user story and feature" - - acceptance_criteria_review: "What must be satisfied" - technical_approach: "How to implement" - component_changes: "What needs to be modified" - api_design: "Interface specifications" - data_model: "Schema changes" - - testing_approach: "How to verify" + - testing: "How to verify; reference governing-story AC-IDs (US-{n}-AC{m})" example: file: example.md diff --git a/workflows/activities/02-design/artifacts/technical-design/prompt.md b/workflows/activities/02-design/artifacts/technical-design/prompt.md index 3ef76ee3..632d98c7 100644 --- a/workflows/activities/02-design/artifacts/technical-design/prompt.md +++ b/workflows/activities/02-design/artifacts/technical-design/prompt.md @@ -9,10 +9,10 @@ unique job is to make one user story buildable by naming the concrete component changes, files, interfaces, data model changes, security implications, tests, rollback path, and implementation sequence. -It inherits Architecture and Solution Design. It must not redesign the feature -or system. If the story cannot be implemented without changing the parent -Solution Design, ADR, Contract, or Architecture, update that governing artifact -first. +It inherits Architecture and Solution Design. For what belongs at this level +versus those higher levels, see the zoom-stack matrix in +`workflows/activities/02-design/README.md`; if the story forces a change at a +higher level, update that governing artifact first. ## Reference Anchors @@ -32,23 +32,15 @@ but leaves no trace here is drift (reconcile-alignment Concern->Artifact Realiza ## Focus - Create a story-level artifact named `docs/helix/02-design/technical-designs/TD-XXX-[name].md`. -- Map each acceptance criterion to component changes, interfaces, data, security, and tests. -- Stay on the vertical slice for the story. -- Assume the broader architecture is already set by the parent solution design. -- Do not expand into a feature-wide or system-wide design; that belongs in a - solution design (`SD-XXX-*`). +- Realize each governing-story AC-ID (US-{n}-AC{m}) through component changes, interfaces, data, security, and tests; reference AC-IDs, do not restate AC text (ADR-009 — AC ownership lives in user-stories). +- Stay on the vertical slice for the story, within the story scope defined in + the zoom-stack matrix (`workflows/activities/02-design/README.md`). - Keep implementation sequence and rollout or migration notes only when they affect execution. ## Boundary Test -| If you are writing... | Put it in... | -|---|---| -| Feature-wide approach or decomposition | Solution Design | -| One architectural decision | ADR | -| Exact external interface contract | Contract | -| One story's implementation shape, files, tests, and rollback | Technical Design | -| Test fixtures and detailed test cases | Story Test Plan | -| Work queue slicing and execution status | Implementation Plan or runtime work item | +See the zoom-stack matrix in `workflows/activities/02-design/README.md` for +which decisions belong at the system, feature, and story levels. ## Completion Criteria - The story is implementable. diff --git a/workflows/activities/02-design/artifacts/technical-design/template.md b/workflows/activities/02-design/artifacts/technical-design/template.md index 796198c5..63ffb5b9 100644 --- a/workflows/activities/02-design/artifacts/technical-design/template.md +++ b/workflows/activities/02-design/artifacts/technical-design/template.md @@ -22,11 +22,6 @@ ddx: - Do not redefine cross-component architecture here; that belongs in `SD-XXX` - Governing artifacts: [User Story, Solution Design, Contracts, Concerns] -## Acceptance Criteria - -1. **Given** [precondition], **When** [action], **Then** [expected outcome] -2. **Given** [precondition], **When** [action], **Then** [expected outcome] - ## Technical Approach **Strategy**: [Brief description] @@ -129,7 +124,7 @@ CREATE TABLE [table_name] ( Use this checklist when reviewing a technical design: -- [ ] Acceptance criteria use Given/When/Then format and are verifiable +- [ ] Each governing-story AC-ID (US-{n}-AC{m}) is realized by the technical changes (AC text is not restated here — ADR-009) - [ ] Technical approach inherits from the parent solution design — no contradictions - [ ] Key decisions have documented rationale - [ ] Trade-offs are explicit — what we gain and what we lose diff --git a/workflows/activities/03-test/artifacts/data-quality-expectations/example.md b/workflows/activities/03-test/artifacts/data-quality-expectations/example.md index 0177f594..5ef5f394 100644 --- a/workflows/activities/03-test/artifacts/data-quality-expectations/example.md +++ b/workflows/activities/03-test/artifacts/data-quality-expectations/example.md @@ -2,7 +2,8 @@ ddx: id: example.data-quality-expectations.customer-360 depends_on: - - example.data-prd.customer-360 + # Previous: example.data-prd.customer-360 — dropped when data-prd + # collapsed into prd as kind: data variant (ADR-008). - example.data-architecture.customer-360 --- diff --git a/workflows/activities/03-test/artifacts/data-quality-expectations/meta.yml b/workflows/activities/03-test/artifacts/data-quality-expectations/meta.yml index 8fb575cb..13d5cf3d 100644 --- a/workflows/activities/03-test/artifacts/data-quality-expectations/meta.yml +++ b/workflows/activities/03-test/artifacts/data-quality-expectations/meta.yml @@ -37,7 +37,7 @@ validation: severity: blocking - check: expectations_trace_to_requirements - description: Each expectation must trace to a data quality requirement in the data-prd + description: Each expectation must trace to a data quality requirement in the PRD (kind: data) severity: blocking - check: failure_modes_explicit @@ -110,7 +110,9 @@ examples: relationships: informed_by: - - data-prd + # `prd` with `kind: data` (per ADR-008) supplies the quality requirements + # this artifact translates into executable expectations. + - prd - data-architecture informs: diff --git a/workflows/activities/03-test/artifacts/data-quality-expectations/prompt.md b/workflows/activities/03-test/artifacts/data-quality-expectations/prompt.md index e1e283f7..804a6da7 100644 --- a/workflows/activities/03-test/artifacts/data-quality-expectations/prompt.md +++ b/workflows/activities/03-test/artifacts/data-quality-expectations/prompt.md @@ -5,7 +5,7 @@ testable predicates on data shape, completeness, accuracy, and freshness. ## Purpose -Data Quality Expectations is the **contract layer between Data PRD (what quality +Data Quality Expectations is the **contract layer between PRD (kind: data) (what quality is needed) and implementation** (how to verify it). Its unique job is to define measurable quality assertions as executable tests: data should not be released to consumers until these contracts are satisfied. @@ -28,7 +28,7 @@ Use these local resource summaries as grounding: ## Focus -- For each requirement in the Data PRD, write at least one testable expectation. +- For each requirement in the PRD (kind: data), write at least one testable expectation. - Group expectations by layer (Bronze, Silver, Gold) and validate at the appropriate layer. - Use SDP `EXPECT` syntax for Databricks Streaming Tables, dbt tests for @@ -55,7 +55,7 @@ platform, substitute as follows: ## Completion Criteria -- Every P0 requirement from Data PRD has at least one corresponding expectation. +- Every P0 requirement from PRD (kind: data) has at least one corresponding expectation. - Expectations are grouped by layer (Bronze input validation, Silver transformation contracts, Gold business rule verification). - Expectations are written in the platform's native syntax (SDP `EXPECT`, diff --git a/workflows/activities/03-test/artifacts/data-quality-expectations/template.md b/workflows/activities/03-test/artifacts/data-quality-expectations/template.md index 5bfbfbf2..00b631cf 100644 --- a/workflows/activities/03-test/artifacts/data-quality-expectations/template.md +++ b/workflows/activities/03-test/artifacts/data-quality-expectations/template.md @@ -7,7 +7,7 @@ ddx: ## Overview and Scope -[Define the data product being tested, the medallion layers covered (Bronze, Silver, Gold), and the quality dimensions in scope (completeness, timeliness, accuracy, consistency, uniqueness). Reference the [[data-prd]] for quality requirements and [[data-architecture]] for the layer definitions. This document translates those requirements into executable EXPECT clauses.] +[Define the data product being tested, the medallion layers covered (Bronze, Silver, Gold), and the quality dimensions in scope (completeness, timeliness, accuracy, consistency, uniqueness). Reference the [[prd]] (kind: data) for quality requirements and [[data-architecture]] for the layer definitions. This document translates those requirements into executable EXPECT clauses.] ### Quality Dimensions @@ -439,7 +439,7 @@ Use this checklist during review to validate that the data quality expectations - [ ] **Silver Layer Expectations** enforce deduplication, nullability, normalization, and referential integrity - [ ] **Gold Layer Expectations** guarantee freshness, consistency, and accuracy for consumer queries - [ ] **Expectations are written in executable form**: SDP EXPECT, dbt test, or SQL constraint (not prose) -- [ ] **Each expectation traces back** to a quality requirement in [[data-prd]] (reference by name) +- [ ] **Each expectation traces back** to a quality requirement in [[prd]] (kind: data) (reference by name) - [ ] **Failure modes are explicit**: What happens when an expectation fails (reject, alert, quarantine)? - [ ] **SLA per layer** is documented: freshness target, detection time, recovery time - [ ] **Sampling vs exhaustive** is clear: All rows tested, or sample with confidence interval documented? diff --git a/workflows/activities/03-test/artifacts/security-tests/security-test.go.template b/workflows/activities/03-test/artifacts/security-tests/security-test.go.template deleted file mode 100644 index 778783af..00000000 --- a/workflows/activities/03-test/artifacts/security-tests/security-test.go.template +++ /dev/null @@ -1,592 +0,0 @@ -package security_test - -import ( - "context" - "fmt" - "strings" - "testing" - "time" - - "github.com/stretchr/testify/assert" - "github.com/stretchr/testify/require" -) - -// Security test suite following OWASP guidelines - -// TestSecurity_InjectionPrevention tests protection against injection attacks -func TestSecurity_InjectionPrevention(t *testing.T) { - tests := []struct { - name string - attackType string - payload string - targetField string - expectedErr bool - description string - }{ - // SQL Injection tests - { - name: "sql_injection_basic", - attackType: "SQL Injection", - payload: "' OR '1'='1", - targetField: "{{sql_field}}", - expectedErr: true, - description: "Should reject basic SQL injection", - }, - { - name: "sql_injection_union", - attackType: "SQL Injection", - payload: "' UNION SELECT * FROM users--", - targetField: "{{sql_field}}", - expectedErr: true, - description: "Should reject UNION-based injection", - }, - { - name: "sql_injection_time_based", - attackType: "SQL Injection", - payload: "'; WAITFOR DELAY '00:00:05'--", - targetField: "{{sql_field}}", - expectedErr: true, - description: "Should reject time-based blind injection", - }, - - // NoSQL Injection tests - { - name: "nosql_injection_mongodb", - attackType: "NoSQL Injection", - payload: `{"$ne": null}`, - targetField: "{{nosql_field}}", - expectedErr: true, - description: "Should reject MongoDB injection", - }, - - // Command Injection tests - { - name: "command_injection_semicolon", - attackType: "Command Injection", - payload: "test; rm -rf /", - targetField: "{{command_field}}", - expectedErr: true, - description: "Should reject command chaining", - }, - { - name: "command_injection_backtick", - attackType: "Command Injection", - payload: "test`whoami`", - targetField: "{{command_field}}", - expectedErr: true, - description: "Should reject backtick command substitution", - }, - - // LDAP Injection tests - { - name: "ldap_injection", - attackType: "LDAP Injection", - payload: "*)(uid=*", - targetField: "{{ldap_field}}", - expectedErr: true, - description: "Should reject LDAP filter injection", - }, - - // XML Injection tests - { - name: "xml_injection_xxe", - attackType: "XML Injection", - payload: `]>`, - targetField: "{{xml_field}}", - expectedErr: true, - description: "Should reject XXE attacks", - }, - } - - for _, tt := range tests { - t.Run(tt.name, func(t *testing.T) { - // Execute injection attempt - err := {{ExecuteWithPayload}}(tt.targetField, tt.payload) - - if tt.expectedErr { - assert.Error(t, err, "%s: %s", tt.attackType, tt.description) - // Verify payload was sanitized or rejected - assert.NotContains(t, err.Error(), "executed", "Payload should not execute") - } else { - assert.NoError(t, err, "Valid input should be accepted") - } - }) - } -} - -// TestSecurity_Authentication tests authentication mechanisms -func TestSecurity_Authentication(t *testing.T) { - tests := []struct { - name string - scenario string - setup func() {{AuthContext}} - expectedErr bool - }{ - { - name: "valid_credentials", - scenario: "Valid username and password", - setup: func() {{AuthContext}} { - return {{CreateValidAuth}}() - }, - expectedErr: false, - }, - { - name: "invalid_password", - scenario: "Incorrect password", - setup: func() {{AuthContext}} { - return {{CreateInvalidPasswordAuth}}() - }, - expectedErr: true, - }, - { - name: "nonexistent_user", - scenario: "User does not exist", - setup: func() {{AuthContext}} { - return {{CreateNonexistentUserAuth}}() - }, - expectedErr: true, - }, - { - name: "expired_token", - scenario: "Expired authentication token", - setup: func() {{AuthContext}} { - return {{CreateExpiredTokenAuth}}() - }, - expectedErr: true, - }, - { - name: "malformed_token", - scenario: "Malformed JWT token", - setup: func() {{AuthContext}} { - return {{CreateMalformedTokenAuth}}() - }, - expectedErr: true, - }, - } - - for _, tt := range tests { - t.Run(tt.name, func(t *testing.T) { - authCtx := tt.setup() - - // Attempt authentication - _, err := {{Authenticate}}(authCtx) - - if tt.expectedErr { - assert.Error(t, err, "Should reject: %s", tt.scenario) - } else { - assert.NoError(t, err, "Should accept: %s", tt.scenario) - } - }) - } -} - -// TestSecurity_Authorization tests access control -func TestSecurity_Authorization(t *testing.T) { - tests := []struct { - name string - user {{UserType}} - resource string - action string - expected bool - description string - }{ - { - name: "admin_full_access", - user: {{CreateAdminUser}}(), - resource: "{{admin_resource}}", - action: "write", - expected: true, - description: "Admin should have full access", - }, - { - name: "user_read_only", - user: {{CreateRegularUser}}(), - resource: "{{user_resource}}", - action: "read", - expected: true, - description: "Regular user should have read access", - }, - { - name: "user_no_write", - user: {{CreateRegularUser}}(), - resource: "{{admin_resource}}", - action: "write", - expected: false, - description: "Regular user should not have write access", - }, - { - name: "privilege_escalation_attempt", - user: {{CreateMaliciousUser}}(), - resource: "{{sensitive_resource}}", - action: "delete", - expected: false, - description: "Should prevent privilege escalation", - }, - } - - for _, tt := range tests { - t.Run(tt.name, func(t *testing.T) { - allowed := {{CheckAuthorization}}(tt.user, tt.resource, tt.action) - assert.Equal(t, tt.expected, allowed, tt.description) - }) - } -} - -// TestSecurity_DataProtection tests encryption and data security -func TestSecurity_DataProtection(t *testing.T) { - t.Run("sensitive_data_encryption", func(t *testing.T) { - // Test that sensitive data is encrypted at rest - sensitiveData := {{CreateSensitiveData}}() - - // Store data - id, err := {{StoreData}}(sensitiveData) - require.NoError(t, err) - - // Retrieve raw data from storage - rawData, err := {{GetRawData}}(id) - require.NoError(t, err) - - // Verify data is encrypted - assert.NotContains(t, string(rawData), sensitiveData.PlainText, - "Sensitive data should not be stored in plain text") - }) - - t.Run("password_hashing", func(t *testing.T) { - password := "SecurePassword123!" - - // Hash password - hashed, err := {{HashPassword}}(password) - require.NoError(t, err) - - // Verify not plain text - assert.NotEqual(t, password, hashed, "Password should be hashed") - - // Verify bcrypt or argon2 format - assert.True(t, strings.HasPrefix(hashed, "$2") || strings.HasPrefix(hashed, "$argon2"), - "Should use secure hashing algorithm") - }) - - t.Run("pii_masking", func(t *testing.T) { - // Test that PII is masked in logs - userData := {{CreateUserWithPII}}() - - // Log user data - logOutput := {{CaptureLogOutput}}(func() { - {{LogUserData}}(userData) - }) - - // Verify PII is masked - assert.NotContains(t, logOutput, userData.SSN, "SSN should be masked") - assert.NotContains(t, logOutput, userData.CreditCard, "Credit card should be masked") - assert.Contains(t, logOutput, "****", "Should contain masked values") - }) -} - -// TestSecurity_SessionManagement tests session security -func TestSecurity_SessionManagement(t *testing.T) { - t.Run("session_timeout", func(t *testing.T) { - // Create session - session := {{CreateSession}}() - - // Verify initial validity - assert.True(t, {{IsSessionValid}}(session), "New session should be valid") - - // Simulate timeout - time.Sleep({{SessionTimeout}} + time.Second) - - // Verify session expired - assert.False(t, {{IsSessionValid}}(session), "Session should expire after timeout") - }) - - t.Run("session_fixation_prevention", func(t *testing.T) { - // Get session before authentication - preAuthSession := {{GetCurrentSession}}() - - // Authenticate - err := {{AuthenticateUser}}("user", "password") - require.NoError(t, err) - - // Get session after authentication - postAuthSession := {{GetCurrentSession}}() - - // Verify session ID changed - assert.NotEqual(t, preAuthSession.ID, postAuthSession.ID, - "Session ID should change after authentication") - }) - - t.Run("concurrent_session_limit", func(t *testing.T) { - user := {{CreateTestUser}}() - - // Create multiple sessions - const maxSessions = {{MaxConcurrentSessions}} - sessions := make([]{{SessionType}}, 0, maxSessions+1) - - for i := 0; i < maxSessions; i++ { - session, err := {{CreateUserSession}}(user) - require.NoError(t, err) - sessions = append(sessions, session) - } - - // Try to create one more session - _, err := {{CreateUserSession}}(user) - assert.Error(t, err, "Should not allow more than %d concurrent sessions", maxSessions) - }) -} - -// TestSecurity_InputValidation tests input sanitization -func TestSecurity_InputValidation(t *testing.T) { - tests := []struct { - name string - input string - validator func(string) error - expectedErr bool - description string - }{ - // Path traversal - { - name: "path_traversal_dots", - input: "../../etc/passwd", - validator: {{ValidateFilePath}}, - expectedErr: true, - description: "Should reject path traversal", - }, - - // XSS prevention - { - name: "xss_script_tag", - input: ``, - validator: {{ValidateHTML}}, - expectedErr: true, - description: "Should reject script tags", - }, - { - name: "xss_event_handler", - input: ``, - validator: {{ValidateHTML}}, - expectedErr: true, - description: "Should reject event handlers", - }, - - // Email validation - { - name: "valid_email", - input: "user@example.com", - validator: {{ValidateEmail}}, - expectedErr: false, - description: "Should accept valid email", - }, - { - name: "invalid_email", - input: "not-an-email", - validator: {{ValidateEmail}}, - expectedErr: true, - description: "Should reject invalid email", - }, - - // URL validation - { - name: "javascript_url", - input: "javascript:alert('XSS')", - validator: {{ValidateURL}}, - expectedErr: true, - description: "Should reject javascript: URLs", - }, - { - name: "data_url", - input: "data:text/html,", - validator: {{ValidateURL}}, - expectedErr: true, - description: "Should reject data: URLs", - }, - } - - for _, tt := range tests { - t.Run(tt.name, func(t *testing.T) { - err := tt.validator(tt.input) - - if tt.expectedErr { - assert.Error(t, err, tt.description) - } else { - assert.NoError(t, err, tt.description) - } - }) - } -} - -// TestSecurity_RateLimiting tests rate limiting and DDoS protection -func TestSecurity_RateLimiting(t *testing.T) { - t.Run("api_rate_limit", func(t *testing.T) { - const maxRequests = {{MaxRequestsPerMinute}} - client := {{CreateTestClient}}() - - // Make requests up to limit - for i := 0; i < maxRequests; i++ { - err := {{MakeAPIRequest}}(client) - assert.NoError(t, err, "Request %d should succeed", i+1) - } - - // Next request should be rate limited - err := {{MakeAPIRequest}}(client) - assert.Error(t, err, "Should be rate limited after %d requests", maxRequests) - assert.Contains(t, err.Error(), "rate limit", "Error should indicate rate limiting") - }) - - t.Run("login_attempt_limit", func(t *testing.T) { - const maxAttempts = {{MaxLoginAttempts}} - username := "testuser" - - // Make failed login attempts - for i := 0; i < maxAttempts; i++ { - err := {{AttemptLogin}}(username, "wrong-password") - assert.Error(t, err, "Login attempt %d should fail", i+1) - } - - // Account should be locked - err := {{AttemptLogin}}(username, "correct-password") - assert.Error(t, err, "Account should be locked after %d failed attempts", maxAttempts) - assert.Contains(t, err.Error(), "locked", "Error should indicate account lock") - }) -} - -// TestSecurity_CSRF tests CSRF protection -func TestSecurity_CSRF(t *testing.T) { - t.Run("csrf_token_required", func(t *testing.T) { - // Make request without CSRF token - req := {{CreatePOSTRequest}}("/api/sensitive-action") - resp := {{ExecuteRequest}}(req) - - assert.Equal(t, 403, resp.StatusCode, "Should reject request without CSRF token") - }) - - t.Run("csrf_token_validation", func(t *testing.T) { - // Get valid CSRF token - token := {{GetCSRFToken}}() - - // Make request with valid token - req := {{CreatePOSTRequest}}("/api/sensitive-action") - req.Header.Set("X-CSRF-Token", token) - resp := {{ExecuteRequest}}(req) - - assert.Equal(t, 200, resp.StatusCode, "Should accept request with valid CSRF token") - }) - - t.Run("csrf_token_invalidation", func(t *testing.T) { - // Get CSRF token - token := {{GetCSRFToken}}() - - // Use token once - req1 := {{CreatePOSTRequest}}("/api/sensitive-action") - req1.Header.Set("X-CSRF-Token", token) - resp1 := {{ExecuteRequest}}(req1) - assert.Equal(t, 200, resp1.StatusCode) - - // Try to reuse token - req2 := {{CreatePOSTRequest}}("/api/another-action") - req2.Header.Set("X-CSRF-Token", token) - resp2 := {{ExecuteRequest}}(req2) - - assert.Equal(t, 403, resp2.StatusCode, "Should reject reused CSRF token") - }) -} - -// TestSecurity_ErrorHandling tests secure error handling -func TestSecurity_ErrorHandling(t *testing.T) { - t.Run("no_sensitive_info_in_errors", func(t *testing.T) { - // Trigger various errors - errors := []error{ - {{TriggerDatabaseError}}(), - {{TriggerFileError}}(), - {{TriggerAuthError}}(), - } - - for _, err := range errors { - errMsg := err.Error() - - // Check for sensitive information - assert.NotContains(t, errMsg, "password", "Error should not contain passwords") - assert.NotContains(t, errMsg, "{{db_connection_string}}", "Error should not contain connection strings") - assert.NotContains(t, errMsg, "{{api_key}}", "Error should not contain API keys") - assert.NotRegexp(t, `\/home\/[\w]+\/`, errMsg, "Error should not contain system paths") - } - }) - - t.Run("generic_error_messages", func(t *testing.T) { - // Test that user-facing errors are generic - userErrors := map[string]func() error{ - "invalid_credentials": func() error { - return {{AuthenticateUser}}("nonexistent", "password") - }, - "unauthorized_access": func() error { - return {{AccessResource}}("{{restricted_resource}}") - }, - } - - for name, triggerError := range userErrors { - t.Run(name, func(t *testing.T) { - err := triggerError() - assert.Error(t, err) - - // Error should be generic - assert.Contains(t, err.Error(), "{{generic_error_message}}", - "User-facing error should be generic") - }) - } - }) -} - -// TestSecurity_Compliance tests regulatory compliance -func TestSecurity_Compliance(t *testing.T) { - t.Run("gdpr_data_deletion", func(t *testing.T) { - // Create user with data - userID := {{CreateUserWithData}}() - - // Request data deletion - err := {{RequestDataDeletion}}(userID) - require.NoError(t, err) - - // Verify data is deleted - data, err := {{GetUserData}}(userID) - assert.Error(t, err, "User data should be deleted") - assert.Nil(t, data, "No data should be returned") - }) - - t.Run("pci_dss_card_storage", func(t *testing.T) { - // Attempt to store full credit card number - cardNumber := "4111111111111111" - err := {{StorePaymentMethod}}(cardNumber) - - // Should either error or store tokenized - if err == nil { - // Verify only last 4 digits stored - stored := {{GetStoredPaymentMethod}}() - assert.Equal(t, "****1111", stored, "Should only store masked card number") - } - }) - - t.Run("hipaa_audit_logging", func(t *testing.T) { - // Access protected health information - {{AccessPHI}}("patient-123") - - // Verify audit log created - auditLogs := {{GetAuditLogs}}() - require.NotEmpty(t, auditLogs) - - lastLog := auditLogs[len(auditLogs)-1] - assert.Contains(t, lastLog, "PHI access", "Should log PHI access") - assert.Contains(t, lastLog, "patient-123", "Should log resource ID") - assert.Contains(t, lastLog, "{{current_user}}", "Should log accessing user") - }) -} - -// Helper function to run security scan -func runSecurityScan(t *testing.T, target string) { - // This would integrate with actual security scanning tools - // For testing, we simulate the scan - results := {{RunSecurityScan}}(target) - - for _, issue := range results.Issues { - if issue.Severity == "HIGH" || issue.Severity == "CRITICAL" { - t.Errorf("Security issue found: %s (Severity: %s)", issue.Description, issue.Severity) - } - } -} \ No newline at end of file diff --git a/workflows/activities/03-test/artifacts/story-test-plan/meta.yml b/workflows/activities/03-test/artifacts/story-test-plan/meta.yml index d6e6372a..75d25b2a 100644 --- a/workflows/activities/03-test/artifacts/story-test-plan/meta.yml +++ b/workflows/activities/03-test/artifacts/story-test-plan/meta.yml @@ -37,7 +37,7 @@ validation: required_sections: - story_reference - scope_and_objective - - acceptance_criteria_mapping + - acceptance_criteria_test_mapping - executable_proof - data_and_setup - build_handoff diff --git a/workflows/activities/03-test/artifacts/story-test-plan/prompt.md b/workflows/activities/03-test/artifacts/story-test-plan/prompt.md index dce2c3ad..ba5ffb24 100644 --- a/workflows/activities/03-test/artifacts/story-test-plan/prompt.md +++ b/workflows/activities/03-test/artifacts/story-test-plan/prompt.md @@ -1,18 +1,11 @@ # Story Test Plan Generation Prompt -Create the canonical story-scoped test plan for one bounded story slice. This -artifact exists because the project-wide `test-plan.md` does not replace the -need for per-story acceptance-to-test traceability. +Scope: one story's acceptance-criteria-to-test traceability — concrete failing tests, fixtures, commands, and setup for a single bounded slice. -## Purpose - -Story Test Plan is the **story-level executable verification handoff**. Its -unique job is to turn one user story's acceptance criteria and technical design -into concrete failing tests, fixtures, commands, and setup before Build starts. - -It inherits the project Test Plan. It does not redefine test strategy, coverage -targets, or feature-wide risk. It owns the exact evidence needed for one story -slice. +Related: +- [Test Plan](../test-plan/prompt.md) — project-wide strategy this STP inherits +- [Test Suites](../test-suites/prompt.md) — where these story tests live under `tests/` +- [Test Procedures](../test-procedures/prompt.md) — how tests get written and run ## Reference Anchors diff --git a/workflows/activities/03-test/artifacts/test-plan/meta.yml b/workflows/activities/03-test/artifacts/test-plan/meta.yml index 8622aeee..1cba6e5c 100644 --- a/workflows/activities/03-test/artifacts/test-plan/meta.yml +++ b/workflows/activities/03-test/artifacts/test-plan/meta.yml @@ -37,9 +37,8 @@ reuse_policy: | validation: required_sections: - testing_strategy - - test_levels - coverage_requirements - - critical_paths + - acceptance_criteria_layer_allocation pattern_checks: - pattern: "P0|P1|P2|Critical" diff --git a/workflows/activities/03-test/artifacts/test-plan/prompt.md b/workflows/activities/03-test/artifacts/test-plan/prompt.md index 806855e4..794daa06 100644 --- a/workflows/activities/03-test/artifacts/test-plan/prompt.md +++ b/workflows/activities/03-test/artifacts/test-plan/prompt.md @@ -1,17 +1,11 @@ # Test Plan Generation Prompt -Create the project-level test plan for the Test activity. Keep it concise, but include the minimum structure needed to drive failing tests before implementation. +Scope: project-level verification strategy — test levels, coverage targets, critical paths, data strategy, infrastructure, sequencing, risks, and build handoff commands. -## Purpose - -The Test Plan is the **project-level verification strategy**. Its unique job is -to define test levels, coverage targets, critical paths, data strategy, -infrastructure, sequencing, risks, and build handoff commands before -implementation starts. - -It does not contain every story-specific test case. Story Test Plans own the -exact executable checks for one story. The Test Plan owns the portfolio: what -must be covered, where confidence should come from, and how CI enforces it. +Related: +- [Story Test Plan](../story-test-plan/prompt.md) — per-story AC↔test traceability +- [Test Suites](../test-suites/prompt.md) — suite inventory and boundaries under `tests/` +- [Test Procedures](../test-procedures/prompt.md) — runner procedures, commands, evidence capture ## Reference Anchors diff --git a/workflows/activities/03-test/artifacts/test-procedures/prompt.md b/workflows/activities/03-test/artifacts/test-procedures/prompt.md index f3c503cc..fffa4cdb 100644 --- a/workflows/activities/03-test/artifacts/test-procedures/prompt.md +++ b/workflows/activities/03-test/artifacts/test-procedures/prompt.md @@ -1,6 +1,11 @@ # Test Procedures Generation Prompt -Create concise procedures for writing, running, and maintaining tests. Focus on the steps an implementer needs for this project, not general testing advice. +Scope: procedures for writing, running, and maintaining tests — per-type writing rules, execution commands, validation checks, and evidence capture. + +Related: +- [Test Plan](../test-plan/prompt.md) — project verification strategy these procedures execute +- [Story Test Plan](../story-test-plan/prompt.md) — per-story tests these procedures apply to +- [Test Suites](../test-suites/prompt.md) — suite layout these procedures operate on ## Reference Anchors diff --git a/workflows/activities/03-test/artifacts/test-suites/integration-test.go.template b/workflows/activities/03-test/artifacts/test-suites/integration-test.go.template deleted file mode 100644 index d233b8d3..00000000 --- a/workflows/activities/03-test/artifacts/test-suites/integration-test.go.template +++ /dev/null @@ -1,348 +0,0 @@ -package {{package_name}}_test - -import ( - "context" - "os" - "testing" - "time" - - "github.com/stretchr/testify/assert" - "github.com/stretchr/testify/require" - "github.com/stretchr/testify/suite" -) - -// {{IntegrationName}}TestSuite tests integration between {{components}} -type {{IntegrationName}}TestSuite struct { - suite.Suite - - // Test resources - {{resource_1}} {{Resource1Type}} - {{resource_2}} {{Resource2Type}} - - // Test context - ctx context.Context - cancel context.CancelFunc - - // Cleanup functions - cleanups []func() -} - -// SetupSuite runs once before all tests -func (s *{{IntegrationName}}TestSuite) SetupSuite() { - // Skip if short tests only - if testing.Short() { - s.T().Skip("Skipping integration tests in short mode") - } - - // Create context with timeout - s.ctx, s.cancel = context.WithTimeout(context.Background(), {{test_timeout}}*time.Minute) - - // Initialize shared resources - s.setup{{Resource1}}() - s.setup{{Resource2}}() - - // Verify connections - s.verifyConnections() -} - -// TearDownSuite runs once after all tests -func (s *{{IntegrationName}}TestSuite) TearDownSuite() { - // Cancel context - if s.cancel != nil { - s.cancel() - } - - // Run cleanup functions in reverse order - for i := len(s.cleanups) - 1; i >= 0; i-- { - if s.cleanups[i] != nil { - s.cleanups[i]() - } - } -} - -// SetupTest runs before each test -func (s *{{IntegrationName}}TestSuite) SetupTest() { - // Reset test data - s.resetTestData() - - // Clear any cached state - s.clearCache() -} - -// TearDownTest runs after each test -func (s *{{IntegrationName}}TestSuite) TearDownTest() { - // Clean up test-specific resources - s.cleanupTestResources() -} - -// Test{{Operation}}_Success tests successful {{operation}} between components -func (s *{{IntegrationName}}TestSuite) Test{{Operation}}_Success() { - // Arrange - testData := s.createTestData("{{test_data_scenario}}") - expectedResult := {{expected_result}} - - // Act - actualResult, err := s.{{resource_1}}.{{Operation}}(s.ctx, testData) - - // Assert - s.Require().NoError(err, "Operation should succeed") - s.Assert().Equal(expectedResult, actualResult, "Result should match expected") - - // Verify side effects - s.verifySideEffects(testData, actualResult) -} - -// Test{{Operation}}_ErrorHandling tests error scenarios -func (s *{{IntegrationName}}TestSuite) Test{{Operation}}_ErrorHandling() { - tests := []struct { - name string - scenario string - setup func() - expectedErr string - }{ - { - name: "invalid_input", - scenario: "Invalid input data", - setup: func() { - // Setup invalid conditions - {{invalid_setup}} - }, - expectedErr: "{{expected_error_1}}", - }, - { - name: "resource_unavailable", - scenario: "Resource temporarily unavailable", - setup: func() { - // Simulate resource unavailability - {{unavailable_setup}} - }, - expectedErr: "{{expected_error_2}}", - }, - { - name: "timeout", - scenario: "Operation timeout", - setup: func() { - // Create short timeout context - {{timeout_setup}} - }, - expectedErr: "context deadline exceeded", - }, - } - - for _, tt := range tests { - s.Run(tt.name, func() { - // Setup error condition - if tt.setup != nil { - tt.setup() - defer s.resetTestData() // Reset after each test - } - - // Execute operation - _, err := s.{{resource_1}}.{{Operation}}(s.ctx, nil) - - // Verify error - s.Require().Error(err, "Should return error for: %s", tt.scenario) - s.Assert().Contains(err.Error(), tt.expectedErr, "Error message mismatch") - }) - } -} - -// Test{{Operation}}_Concurrent tests concurrent operations -func (s *{{IntegrationName}}TestSuite) Test{{Operation}}_Concurrent() { - const numWorkers = {{num_workers}} - const numOperations = {{num_operations}} - - // Channel for collecting results - results := make(chan error, numWorkers*numOperations) - - // Launch concurrent workers - for i := 0; i < numWorkers; i++ { - go func(workerID int) { - for j := 0; j < numOperations; j++ { - // Create unique test data for each operation - testData := s.createTestDataWithID(workerID, j) - - // Execute operation - _, err := s.{{resource_1}}.{{Operation}}(s.ctx, testData) - results <- err - } - }(i) - } - - // Collect results - successCount := 0 - errorCount := 0 - - for i := 0; i < numWorkers*numOperations; i++ { - err := <-results - if err == nil { - successCount++ - } else { - errorCount++ - s.T().Logf("Operation error: %v", err) - } - } - - // Verify results - s.Assert().Greater(successCount, 0, "At least some operations should succeed") - s.Assert().Less(errorCount, numWorkers*numOperations/2, "Error rate should be < 50%") -} - -// Test{{Operation}}_Performance tests performance requirements -func (s *{{IntegrationName}}TestSuite) Test{{Operation}}_Performance() { - // Performance requirements - const maxLatency = {{max_latency_ms}} * time.Millisecond - const minThroughput = {{min_throughput_ops}} - - // Warm up - for i := 0; i < 10; i++ { - testData := s.createTestData("warmup") - s.{{resource_1}}.{{Operation}}(s.ctx, testData) - } - - // Measure latency - start := time.Now() - testData := s.createTestData("performance") - _, err := s.{{resource_1}}.{{Operation}}(s.ctx, testData) - latency := time.Since(start) - - s.Require().NoError(err, "Operation should succeed") - s.Assert().LessOrEqual(latency, maxLatency, "Latency exceeds maximum: %v > %v", latency, maxLatency) - - // Measure throughput - operations := 0 - timeout := time.After(1 * time.Second) - start = time.Now() - -loop: - for { - select { - case <-timeout: - break loop - default: - testData := s.createTestData("throughput") - if _, err := s.{{resource_1}}.{{Operation}}(s.ctx, testData); err == nil { - operations++ - } - } - } - - duration := time.Since(start) - throughput := float64(operations) / duration.Seconds() - - s.Assert().GreaterOrEqual(throughput, float64(minThroughput), - "Throughput below minimum: %.2f ops/s < %d ops/s", throughput, minThroughput) -} - -// Test{{Operation}}_DataIntegrity verifies data consistency -func (s *{{IntegrationName}}TestSuite) Test{{Operation}}_DataIntegrity() { - // Create test data with known values - testData := s.createTestDataWithIntegrity() - - // Perform operation - result, err := s.{{resource_1}}.{{Operation}}(s.ctx, testData) - s.Require().NoError(err) - - // Verify data integrity - s.verifyDataIntegrity(testData, result) - - // Verify data persistence - persistedData, err := s.{{resource_2}}.Get(s.ctx, result.ID) - s.Require().NoError(err) - s.Assert().Equal(result, persistedData, "Persisted data should match result") -} - -// Helper methods - -func (s *{{IntegrationName}}TestSuite) setup{{Resource1}}() { - // Initialize resource 1 - {{resource1_init}} - - // Add cleanup - s.cleanups = append(s.cleanups, func() { - {{resource1_cleanup}} - }) -} - -func (s *{{IntegrationName}}TestSuite) setup{{Resource2}}() { - // Initialize resource 2 - {{resource2_init}} - - // Add cleanup - s.cleanups = append(s.cleanups, func() { - {{resource2_cleanup}} - }) -} - -func (s *{{IntegrationName}}TestSuite) verifyConnections() { - // Verify resource 1 connection - err := s.{{resource_1}}.Ping(s.ctx) - s.Require().NoError(err, "Resource 1 should be accessible") - - // Verify resource 2 connection - err = s.{{resource_2}}.Health(s.ctx) - s.Require().NoError(err, "Resource 2 should be healthy") -} - -func (s *{{IntegrationName}}TestSuite) resetTestData() { - // Reset test data to known state - {{reset_data}} -} - -func (s *{{IntegrationName}}TestSuite) clearCache() { - // Clear any cached data - {{clear_cache}} -} - -func (s *{{IntegrationName}}TestSuite) cleanupTestResources() { - // Clean up test-specific resources - {{cleanup_resources}} -} - -func (s *{{IntegrationName}}TestSuite) createTestData(scenario string) {{TestDataType}} { - // Create test data for scenario - {{create_test_data}} - return testData -} - -func (s *{{IntegrationName}}TestSuite) createTestDataWithID(workerID, operationID int) {{TestDataType}} { - // Create unique test data - {{create_unique_data}} - return testData -} - -func (s *{{IntegrationName}}TestSuite) createTestDataWithIntegrity() {{TestDataType}} { - // Create test data with checksums/hashes for integrity verification - {{create_integrity_data}} - return testData -} - -func (s *{{IntegrationName}}TestSuite) verifySideEffects(input {{TestDataType}}, result {{ResultType}}) { - // Verify expected side effects occurred - {{verify_side_effects}} -} - -func (s *{{IntegrationName}}TestSuite) verifyDataIntegrity(input {{TestDataType}}, result {{ResultType}}) { - // Verify data integrity constraints - {{verify_integrity}} -} - -// TestRun{{IntegrationName}}Suite runs the integration test suite -func TestRun{{IntegrationName}}Suite(t *testing.T) { - suite.Run(t, new({{IntegrationName}}TestSuite)) -} - -// Environment check -func init() { - // Check for required environment variables - requiredEnvVars := []string{ - "{{env_var_1}}", - "{{env_var_2}}", - } - - for _, envVar := range requiredEnvVars { - if os.Getenv(envVar) == "" { - panic("Missing required environment variable: " + envVar) - } - } -} \ No newline at end of file diff --git a/workflows/activities/03-test/artifacts/test-suites/prompt.md b/workflows/activities/03-test/artifacts/test-suites/prompt.md index 72509fdd..8a45ae4a 100644 --- a/workflows/activities/03-test/artifacts/test-suites/prompt.md +++ b/workflows/activities/03-test/artifacts/test-suites/prompt.md @@ -1,6 +1,11 @@ # Test Suites Generation Prompt -Create the test suite layout for the Red activity. Keep it concise and project-specific: define the suite boundaries, the minimum behavior each suite must cover, and any shared data conventions needed to make the tests executable. +Scope: suite layout under `tests/` — boundaries (contract/integration/unit/E2E), behaviors each suite owns, shared fixtures, and execution commands. + +Related: +- [Test Plan](../test-plan/prompt.md) — project verification strategy these suites realize +- [Story Test Plan](../story-test-plan/prompt.md) — per-story tests that land in these suites +- [Test Procedures](../test-procedures/prompt.md) — writing, running, and maintaining the tests in these suites ## Reference Anchors diff --git a/workflows/activities/03-test/artifacts/test-suites/unit-test.go.template b/workflows/activities/03-test/artifacts/test-suites/unit-test.go.template deleted file mode 100644 index aa0814b9..00000000 --- a/workflows/activities/03-test/artifacts/test-suites/unit-test.go.template +++ /dev/null @@ -1,219 +0,0 @@ -package {{package_name}} - -import ( - "testing" - - "github.com/stretchr/testify/assert" - "github.com/stretchr/testify/require" -) - -// Test{{ComponentName}}_{{TestScenario}} tests {{description}} -func Test{{ComponentName}}_{{TestScenario}}(t *testing.T) { - // Table-driven tests for comprehensive coverage - tests := []struct { - name string - description string - input {{InputType}} - setup func(t *testing.T) {{SetupReturnType}} - expected {{ExpectedType}} - expectError bool - validate func(t *testing.T, actual {{ActualType}}, err error) - }{ - { - name: "{{test_case_1_name}}", - description: "{{test_case_1_description}}", - input: {{test_case_1_input}}, - setup: func(t *testing.T) {{SetupReturnType}} { - // Arrange: Set up test conditions - {{setup_code}} - }, - expected: {{test_case_1_expected}}, - expectError: false, - validate: func(t *testing.T, actual {{ActualType}}, err error) { - // Additional validation beyond simple equality - {{validation_code}} - }, - }, - { - name: "{{test_case_2_name}}", - description: "{{test_case_2_description}}", - input: {{test_case_2_input}}, - setup: func(t *testing.T) {{SetupReturnType}} { - {{setup_code_2}} - }, - expected: {{test_case_2_expected}}, - expectError: {{expect_error}}, - }, - // Edge cases - { - name: "handles_nil_input", - description: "Should handle nil input gracefully", - input: nil, - expectError: true, - validate: func(t *testing.T, actual {{ActualType}}, err error) { - assert.Error(t, err) - assert.Contains(t, err.Error(), "{{expected_error_message}}") - }, - }, - { - name: "handles_empty_input", - description: "Should handle empty input correctly", - input: {{empty_input}}, - expected: {{empty_expected}}, - expectError: false, - }, - { - name: "handles_large_input", - description: "Should handle large input efficiently", - input: {{large_input}}, - setup: func(t *testing.T) {{SetupReturnType}} { - // Create large test data - {{large_data_setup}} - }, - validate: func(t *testing.T, actual {{ActualType}}, err error) { - // Verify performance constraints - {{performance_validation}} - }, - }, - } - - for _, tt := range tests { - t.Run(tt.name, func(t *testing.T) { - // Arrange - var setupResult {{SetupReturnType}} - if tt.setup != nil { - setupResult = tt.setup(t) - } - - // Act - actual, err := {{FunctionUnderTest}}(tt.input) - - // Assert - if tt.expectError { - assert.Error(t, err, "Expected error for: %s", tt.description) - } else { - assert.NoError(t, err, "Unexpected error for: %s", tt.description) - if tt.expected != nil { - assert.Equal(t, tt.expected, actual, "Mismatch for: %s", tt.description) - } - } - - // Custom validation - if tt.validate != nil { - tt.validate(t, actual, err) - } - - // Cleanup if needed - if setupResult != nil { - // Cleanup code - {{cleanup_code}} - } - }) - } -} - -// Test{{ComponentName}}_Benchmarks benchmarks performance-critical operations -func Benchmark{{ComponentName}}_{{Operation}}(b *testing.B) { - // Setup benchmark data - {{benchmark_setup}} - - b.ResetTimer() - for i := 0; i < b.N; i++ { - // Benchmark the operation - {{benchmark_operation}} - } -} - -// Test{{ComponentName}}_Concurrent tests thread safety -func Test{{ComponentName}}_Concurrent(t *testing.T) { - if testing.Short() { - t.Skip("Skipping concurrent test in short mode") - } - - // Number of concurrent operations - const numGoroutines = {{num_goroutines}} - const numOperations = {{num_operations}} - - // Setup shared resource - {{shared_resource_setup}} - - // Use WaitGroup for synchronization - var wg sync.WaitGroup - wg.Add(numGoroutines) - - // Track errors from goroutines - errors := make(chan error, numGoroutines*numOperations) - - // Launch concurrent operations - for i := 0; i < numGoroutines; i++ { - go func(id int) { - defer wg.Done() - for j := 0; j < numOperations; j++ { - // Perform concurrent operation - err := {{concurrent_operation}} - if err != nil { - errors <- err - } - } - }(i) - } - - // Wait for all goroutines to complete - wg.Wait() - close(errors) - - // Check for any errors - for err := range errors { - t.Errorf("Concurrent operation failed: %v", err) - } - - // Verify final state - {{concurrent_validation}} -} - -// Test{{ComponentName}}_Fuzz tests with random inputs -func Fuzz{{ComponentName}}(f *testing.F) { - // Add seed corpus - f.Add({{seed_value_1}}) - f.Add({{seed_value_2}}) - f.Add({{seed_value_3}}) - - f.Fuzz(func(t *testing.T, input {{FuzzInputType}}) { - // Execute function with fuzzed input - result, err := {{FunctionUnderTest}}(input) - - // Verify invariants that should always hold - if err == nil { - // Property-based assertions - {{property_assertions}} - } - - // Function should not panic - assert.NotPanics(t, func() { - {{FunctionUnderTest}}(input) - }) - }) -} - -// Helper functions - -func setup{{ComponentName}}TestEnvironment(t *testing.T) func() { - // Setup test environment - {{environment_setup}} - - // Return cleanup function - return func() { - {{environment_cleanup}} - } -} - -func create{{ComponentName}}TestData(t *testing.T) {{TestDataType}} { - // Create test data - {{test_data_creation}} - return testData -} - -func assert{{ComponentName}}State(t *testing.T, expected, actual {{StateType}}) { - // Custom assertion for complex state - {{state_assertion}} -} \ No newline at end of file diff --git a/workflows/activities/05-deploy/artifacts/deployment-checklist/meta.yml b/workflows/activities/05-deploy/artifacts/deployment-checklist/meta.yml index 5bb5b26a..adee3403 100644 --- a/workflows/activities/05-deploy/artifacts/deployment-checklist/meta.yml +++ b/workflows/activities/05-deploy/artifacts/deployment-checklist/meta.yml @@ -21,7 +21,7 @@ validation: - rollout_plan - verification_checks - rollback_triggers - - go_no_go + - go_or_no_go_decision quality_checks: - check: service_specific_steps diff --git a/workflows/activities/05-deploy/artifacts/monitoring-setup/example.md b/workflows/activities/05-deploy/artifacts/monitoring-setup/example.md index c8442f8b..75e87be2 100644 --- a/workflows/activities/05-deploy/artifacts/monitoring-setup/example.md +++ b/workflows/activities/05-deploy/artifacts/monitoring-setup/example.md @@ -93,17 +93,10 @@ ddx: - Import/review SLO burn above 25% in a day triggers rollout hold and owner review. -## Incident Response +## Alert Routing -### Response Entry Points -- Primary owner or schedule: DepositMatch on-call engineer. -- Secondary owner or schedule: platform lead. -- Immediate containment actions: disable import, pause review decisions, or - disable review-log export through feature flags. -- Existing runbook links: `docs/helix/05-deploy/runbook.md`. - -### Routing -- Primary: application on-call. +- Primary: application on-call (DepositMatch on-call engineer). - Secondary: platform lead. - Escalation: product owner for pilot-impact decisions; security lead for data exposure or control failure. +- Runbook entry point: `docs/helix/05-deploy/runbook.md`. diff --git a/workflows/activities/05-deploy/artifacts/monitoring-setup/meta.yml b/workflows/activities/05-deploy/artifacts/monitoring-setup/meta.yml index bfef7142..1b54dca2 100644 --- a/workflows/activities/05-deploy/artifacts/monitoring-setup/meta.yml +++ b/workflows/activities/05-deploy/artifacts/monitoring-setup/meta.yml @@ -22,7 +22,7 @@ validation: - logs_and_tracing - health_checks - sli_slo_tracking - - incident_response + - alert_routing quality_checks: - check: service_specific_signals @@ -34,8 +34,8 @@ validation: - check: health_checks_linked description: Health checks should connect to rollout safety and operator verification severity: warning - - check: incident_response_owned - description: Incident response should name escalation ownership and link a runbook entrypoint once one exists + - check: alert_routing_owned + description: Alert routing should name primary, secondary, and escalation paths and link a runbook entrypoint once one exists severity: warning prompts: diff --git a/workflows/activities/05-deploy/artifacts/monitoring-setup/template.md b/workflows/activities/05-deploy/artifacts/monitoring-setup/template.md index 29b1dcf1..585b1635 100644 --- a/workflows/activities/05-deploy/artifacts/monitoring-setup/template.md +++ b/workflows/activities/05-deploy/artifacts/monitoring-setup/template.md @@ -62,15 +62,9 @@ ddx: ### Error Budget - [Budget and escalation thresholds] -## Incident Response - -### Response Entry Points -- Primary owner or schedule: [Who handles the first page] -- Secondary owner or schedule: [Who backs up the first responder] -- Immediate containment actions: [Safe first actions before deeper procedures exist] -- Existing runbook links (optional): [Add links once a runbook exists] - -### Routing -- Primary: [Schedule] -- Secondary: [Schedule] -- Escalation: [Manager] +## Alert Routing + +- Primary: [Schedule receiving first page] +- Secondary: [Backup schedule] +- Escalation: [Manager or coordinator path] +- Runbook entry point: [Link to runbook once it exists] diff --git a/workflows/activities/06-iterate/README.md b/workflows/activities/06-iterate/README.md index 23ec26a1..a5889e9d 100644 --- a/workflows/activities/06-iterate/README.md +++ b/workflows/activities/06-iterate/README.md @@ -68,6 +68,30 @@ graph TD K -->|No| M[Project Complete] ``` +## Metric Four-Way Slice + +Four iterate artifacts form one slice of the metric loop. Each owns a distinct +job; together they carry an iteration from "what we measure" to "what we do +next": + +- **Metric Definition** is the contract. It fixes the name, unit, direction, + command, and tolerance for one measurement. Every other artifact in the + slice cites a metric definition rather than redefining one. +- **Metrics Dashboard** is the current-values view. It consumes metric + definitions, compares current readings against baseline or ratchet floor, + and produces an iteration-level decision (improved, regressed, noise). +- **Security Metrics** is the security slice of the dashboard. It uses the + same definition contract but scopes the view to incident response, + vulnerability management, application security, and compliance signals. +- **Improvement Backlog** consumes dashboard signal (including the security + slice) and turns it into prioritized follow-up work with an explicit + next-iteration selection. + +Flow: definitions feed dashboards; the security dashboard is the +security-shaped slice of the same data; backlog consumes dashboard signal. +Each artifact prompt cross-references this section instead of restating the +relationship. + ## Work Items ### Artifacts diff --git a/workflows/activities/06-iterate/artifacts/improvement-backlog/prompt.md b/workflows/activities/06-iterate/artifacts/improvement-backlog/prompt.md index f95e7f21..a849f283 100644 --- a/workflows/activities/06-iterate/artifacts/improvement-backlog/prompt.md +++ b/workflows/activities/06-iterate/artifacts/improvement-backlog/prompt.md @@ -9,10 +9,14 @@ unique job is to convert metrics, feedback, incidents, and retrospective learnings into ordered improvement candidates with evidence, rationale, tracker or explicit follow-up targets, and a next-iteration selection. -It is not the live tracker. DDx or another runtime owns issue status, assignees, -and execution history. This artifact explains what should compete for attention +It is not the live tracker. The runtime owns issue status, assignees, and +execution history. This artifact explains what should compete for attention next and why. +For how this artifact relates to metric definitions, the metrics dashboard, +and security-metrics, see the "Metric Four-Way Slice" section of +`workflows/activities/06-iterate/README.md`. + ## Reference Anchors Use this local resource summary as grounding: diff --git a/workflows/activities/06-iterate/artifacts/metric-definition/prompt.md b/workflows/activities/06-iterate/artifacts/metric-definition/prompt.md index 994b31f4..63efe3c2 100644 --- a/workflows/activities/06-iterate/artifacts/metric-definition/prompt.md +++ b/workflows/activities/06-iterate/artifacts/metric-definition/prompt.md @@ -9,8 +9,9 @@ define exactly what is measured, how to collect it, what unit it uses, whether higher or lower is better, what tolerance applies, and how dashboards, ratchets, experiments, or monitoring should interpret it. -It is not a dashboard, alert policy, or improvement backlog item. Those artifacts -consume metric definitions. +For how this artifact relates to dashboards, security-metrics, and the +improvement backlog, see the "Metric Four-Way Slice" section of +`workflows/activities/06-iterate/README.md`. ## Reference Anchors diff --git a/workflows/activities/06-iterate/artifacts/metrics-dashboard/meta.yml b/workflows/activities/06-iterate/artifacts/metrics-dashboard/meta.yml index e1f44970..0f7a3b50 100644 --- a/workflows/activities/06-iterate/artifacts/metrics-dashboard/meta.yml +++ b/workflows/activities/06-iterate/artifacts/metrics-dashboard/meta.yml @@ -17,10 +17,10 @@ output: validation: required_sections: - - review_window - decision - summary - metrics_table + - interpretation_rules - trend_notes - follow_up diff --git a/workflows/activities/06-iterate/artifacts/metrics-dashboard/prompt.md b/workflows/activities/06-iterate/artifacts/metrics-dashboard/prompt.md index d2f7c3aa..b533af5e 100644 --- a/workflows/activities/06-iterate/artifacts/metrics-dashboard/prompt.md +++ b/workflows/activities/06-iterate/artifacts/metrics-dashboard/prompt.md @@ -10,9 +10,9 @@ job is to compare current metric values against explicit baselines, interpret direction and tolerance, and produce a clear decision about improvement, regression, or noise. -It consumes Metric Definitions. It does not redefine metric formulas, command -semantics, or labels. It informs the Improvement Backlog but does not decide -implementation work by itself. +For how this artifact relates to metric definitions, security-metrics, and the +improvement backlog, see the "Metric Four-Way Slice" section of +`workflows/activities/06-iterate/README.md`. ## Reference Anchors diff --git a/workflows/activities/06-iterate/artifacts/security-metrics/prompt.md b/workflows/activities/06-iterate/artifacts/security-metrics/prompt.md index e49bb902..4b69e5a2 100644 --- a/workflows/activities/06-iterate/artifacts/security-metrics/prompt.md +++ b/workflows/activities/06-iterate/artifacts/security-metrics/prompt.md @@ -2,6 +2,10 @@ Create a security metrics report for one iteration. +For how this artifact relates to metric definitions, the metrics dashboard, +and the improvement backlog, see the "Metric Four-Way Slice" section of +`workflows/activities/06-iterate/README.md`. + ## Required Inputs - Security monitoring and incident data for the iteration period - Vulnerability scan results (SAST, DAST, dependency scans) diff --git a/workflows/concerns/README-auth-family.md b/workflows/concerns/README-auth-family.md new file mode 100644 index 00000000..0fbee17d --- /dev/null +++ b/workflows/concerns/README-auth-family.md @@ -0,0 +1,69 @@ +# Auth family — ownership table + +The auth family is a five-way cluster of concerns whose boundaries are easy to +restate and easy to confuse. Per ADR-006 (concern boundary lives once, in +`concern.md`), each concern in the family states its own boundary in its own +`concern.md`. This README is the **single ownership table** the family's +concerns cross-reference, so the per-concern files do not restate each other's +scope. + +Two cross-tree neighbors (`admin-console`, `unity-catalog`) are included +because they materially compose with the family but live outside it. + +## Family members and what each owns + +| Concern | Owns | +|---|---| +| `auth` | principal / session / account bootstrap; **session-token semantics** (issuance, rotation, revocation) | +| `authorization-model` | permission semantics (RBAC / ABAC / ReBAC) and the deny-by-default per-handler check | +| `multi-tenancy` | tenant predicate; tenant scoping rules (the inter-tenant guarantee) | +| `security-owasp` | hardening posture; **audit-logging policy** (what to log on authz denial) | + +The `auth-provider` slot (default filler: `auth-local-sessions`) sits beneath +`auth` and owns the *mechanism* (how identities/sessions are stored and +verified). A different slot filler (Auth0/OIDC, Clerk, …) is a swap, not a +rewrite. The slot is what `auth` defers to; it is not itself a member of the +ownership table because it has no scope `auth` does not already cover at the +surface level. + +## Neighbors that reference into the family + +| Neighbor | References | +|---|---| +| `admin-console` | `auth` (for operator-workflow gates — the operator surface is what `auth` gates) | +| `unity-catalog` | `authorization-model` (Databricks catalog grants compose with app-layer authz; neither substitutes for the other) | + +## Ordering invariants the family preserves + +1. **Authentication precedes authorization.** `auth` establishes the principal; + `authorization-model` decides what that principal may do. The two are + composable and ordered; an authz check on no principal is undefined. +2. **Tenant predicate precedes permission.** `multi-tenancy` guarantees the + record is in the caller's tenant; `authorization-model` then decides whether + the caller may act on it. A permission check on a wrong-tenant record is + still a cross-tenant leak. +3. **Hardening is a posture, not a substitute.** `security-owasp` owns the + umbrella (Broken Access Control, injection, CSRF, secret handling) plus + audit-logging policy on authz denial. It does not replace the per-handler + permission check (`authorization-model`) or the tenant predicate + (`multi-tenancy`). + +## Where ambiguous items land + +Two items historically fell through the gaps between these concerns; this +README records the owner so reviewers do not have to re-litigate: + +- **Session-token semantics** (issuance, rotation, revocation) → `auth`. + Tokens are the materialization of the session `auth` already owns. +- **Audit-logging policy** (what to log on authz denial, login failure, + privilege escalation) → `security-owasp`. Logging policy is a hardening + posture; the authz decision itself is `authorization-model`'s. + +## How to use this README + +- A concern in the family states **only its own** boundary in its `concern.md`. +- Where a concern would otherwise restate a neighbor's scope, it points here + ("see `workflows/concerns/README-auth-family.md` for the family ownership + table") and lets the table do the cross-reference work. +- A new concern that overlaps the family adds itself to the ownership table + in this README rather than restating the table in its own boundary. diff --git a/workflows/concerns/a11y-wcag-aa/concern.md b/workflows/concerns/a11y-wcag-aa/concern.md index 1e0bbc02..c47e0e37 100644 --- a/workflows/concerns/a11y-wcag-aa/concern.md +++ b/workflows/concerns/a11y-wcag-aa/concern.md @@ -32,5 +32,3 @@ Selecting this concern requires these artifacts to change (a selected concern ab - ADR: target WCAG 2.1 AA standard + axe-core + manual screen-reader testing - DESIGN_SYSTEM: keyboard-navigable controls, AA contrast ratios, focus management, labeled inputs - TEST_PLAN: axe-core automated checks + manual screen-reader passes - -## ADR References diff --git a/workflows/concerns/admin-console/concern.md b/workflows/concerns/admin-console/concern.md index 36425407..c27adb19 100644 --- a/workflows/concerns/admin-console/concern.md +++ b/workflows/concerns/admin-console/concern.md @@ -10,14 +10,14 @@ ui, api This concern owns the **operator product surface and its workflow-coverage target**: for an operator-facing product, the human operator can do their -jobs-to-be-done from the UI. It is composable and stays out of four neighbours: - -- **`verification`** owns *evidence* (work isn't done without an observed run). -- **`e2e-framework`** (slot) owns the *tool/form* of the e2e (browser runner, or - live-server HTTP for server-rendered — `verification` decides which). -- **`ux-radix`** owns *interaction quality* (active-state, focus, states). -- **`frontend-framework`** (slot) owns the *stack*; `security-owasp` the - hardening; `auth` gates access. +jobs-to-be-done from the UI. It is composable. + +For access gating, `admin-console` references `auth` — the operator surface is +what `auth` gates; see [README-auth-family.md](../README-auth-family.md) for +the auth family ownership table. For evidence (`verification`), the e2e +tool/form (`e2e-framework` slot), interaction quality (`ux-radix`), the stack +(`frontend-framework` slot), and hardening (`security-owasp`), see those +concerns directly. `admin-console` owns the one thing those do not state: **the operator's jobs-to-be-done — CRUD over the core domain objects plus the domain's diff --git a/workflows/concerns/admin-console/practices.md b/workflows/concerns/admin-console/practices.md index 441eab0f..bde4b26b 100644 --- a/workflows/concerns/admin-console/practices.md +++ b/workflows/concerns/admin-console/practices.md @@ -2,9 +2,11 @@ These practices realize the `admin-console` concern: the operator's jobs-to-be-done built as usable UI, with the primary operator workflow exercised -end-to-end through that UI. They do not restate evidence rules (`verification`), -the e2e tool (`e2e-framework`), interaction quality (`ux-radix`), the stack -(`frontend-framework`), hardening (`security-owasp`), or access gating (`auth`). +end-to-end through that UI. Access gating defers to `auth` — see +[README-auth-family.md](../README-auth-family.md) for the auth family ownership +table. Evidence rules (`verification`), the e2e tool (`e2e-framework`), +interaction quality (`ux-radix`), the stack (`frontend-framework`), and +hardening (`security-owasp`) each live in their own concerns. ## Design diff --git a/workflows/concerns/auth-local-sessions/concern.md b/workflows/concerns/auth-local-sessions/concern.md index 1be410fe..3b2b823c 100644 --- a/workflows/concerns/auth-local-sessions/concern.md +++ b/workflows/concerns/auth-local-sessions/concern.md @@ -13,11 +13,13 @@ auth-provider This concern is the **default `auth-provider` slot filler** — a real, working, self-contained authentication backend that needs no external identity service. -It supplies the *mechanism* the `auth` concern's product surface sits on. It -does not define the surface (signup/login/roles UX — that is `auth`) and does -not own security hardening (that is `security-owasp`). An external IdP -(Auth0/OIDC, Clerk, …) is a **different filler of the same slot**, swapped in -without changing call sites. +It supplies the *mechanism* the `auth` concern's product surface sits on. An +external IdP (Auth0/OIDC, Clerk, …) is a **different filler of the same slot**, +swapped in without changing call sites. + +For the family ownership table (and the boundaries with `auth`, +`authorization-model`, `multi-tenancy`, and `security-owasp`) see +[README-auth-family.md](../README-auth-family.md). ## Components diff --git a/workflows/concerns/auth-local-sessions/practices.md b/workflows/concerns/auth-local-sessions/practices.md index cfefa21a..3456660d 100644 --- a/workflows/concerns/auth-local-sessions/practices.md +++ b/workflows/concerns/auth-local-sessions/practices.md @@ -2,14 +2,20 @@ These practices realize the default `auth-provider` filler — a working local auth backend behind the provider interface the `auth` concern requires. They -cover the *mechanism*; the surface/roles/isolation requirements are in `auth`, -and hardening depth is in `security-owasp`. +cover the *mechanism*; for the family ownership table (auth / +authorization-model / multi-tenancy / security-owasp) see +[README-auth-family.md](../README-auth-family.md). -## Implementation +## Design - **Provider interface**: expose one auth/identity interface (`signup`, `authenticate`, `resolve_principal`, `logout`) that the app calls; keep local-session internals behind it. +- **Swap path**: document how an external IdP filler (Auth0/OIDC) replaces this + one — same interface, config-selected, no call-site change. + +## Build + - **Passwords**: hash with a salted, work-factored algorithm (PBKDF2/bcrypt/ argon2); never store or log plaintext; verify in constant time. - **Sessions**: issue a server-side session on login, referenced by an HttpOnly @@ -17,10 +23,8 @@ and hardening depth is in `security-owasp`. logout. - **Principal resolution**: `resolve_principal` returns the authenticated principal (+ account/tenant + role) for the request, or unauthenticated. -- **Swap path**: document how an external IdP filler (Auth0/OIDC) replaces this - one — same interface, config-selected, no call-site change. -## Quality Gates +## Test - Passwords are hashed (salted + work-factored); no plaintext anywhere. - Session cookie is HttpOnly; sessions validated server-side; logout clears them. diff --git a/workflows/concerns/auth/concern.md b/workflows/concerns/auth/concern.md index cb4d88ae..ddb61add 100644 --- a/workflows/concerns/auth/concern.md +++ b/workflows/concerns/auth/concern.md @@ -10,21 +10,17 @@ api, data, ui This concern owns the **authentication & account product surface** — that an account/multi-tenant product can actually be signed into and is scoped to its -principals. It is composable and does NOT fill a slot. It defers the *backend* -to the `auth-provider` slot, and stays out of three neighbours: - -- **`auth-provider`** (the slot, default `auth-local-sessions`) owns the - *mechanism* — how identities/sessions are stored and verified, and which - provider (local sessions, Auth0/OIDC, Clerk, …) is wired in. `auth` requires - the surface; the slot filler supplies the backend. -- **`security-owasp`** owns *security hardening* — injection, CSRF, secret - handling, password storage strength, rate limiting. `auth` requires that - isolation and roles are enforced; `security-owasp` governs how safely. -- **`admin-console`** owns the *operator workflows*; `auth` is what gates them. - -`auth` owns the one thing those do not state: **an account/tenant product is -not done until a real principal can sign up, sign in, and act only within their -authorized scope.** +principals, plus **session-token semantics** (issuance, rotation, revocation). +It is composable and does NOT fill a slot; it defers the *backend* to the +`auth-provider` slot (default `auth-local-sessions`). + +For the family ownership table (auth / authorization-model / multi-tenancy / +security-owasp, plus the admin-console and unity-catalog neighbors) see +[README-auth-family.md](../README-auth-family.md). + +`auth` owns the one thing the rest of the family does not state: **an +account/tenant product is not done until a real principal can sign up, sign in, +and act only within their authorized scope.** ## Components diff --git a/workflows/concerns/auth/practices.md b/workflows/concerns/auth/practices.md index e220ee7d..206cf873 100644 --- a/workflows/concerns/auth/practices.md +++ b/workflows/concerns/auth/practices.md @@ -2,9 +2,10 @@ These practices realize the `auth` concern: a real, usable authentication & account surface, with authorization and isolation enforced through the -authenticated principal, and the backend behind the `auth-provider` slot. They -do not restate the provider mechanism (that is the slot filler's, e.g. -`auth-local-sessions`) or security hardening (that is `security-owasp`). +authenticated principal, and the backend behind the `auth-provider` slot. For +the family ownership table (and what `security-owasp`, `authorization-model`, +`multi-tenancy`, and the `auth-provider` slot own instead of `auth`) see +[README-auth-family.md](../README-auth-family.md). ## Design diff --git a/workflows/concerns/authorization-model/concern.md b/workflows/concerns/authorization-model/concern.md index d619a0fb..7ad7289c 100644 --- a/workflows/concerns/authorization-model/concern.md +++ b/workflows/concerns/authorization-model/concern.md @@ -23,39 +23,19 @@ This concern owns the **permission model and its enforcement** — *what* an authenticated principal may do to the resources they can legitimately reach, and the discipline that every state-changing and data-returning handler asks that question and refuses by default. It is composable and does **not** fill a slot. -Three neighbours stay distinct, and this concern must not duplicate them: - -- **`auth`** owns **who you are** — signup, login, sessions, and that a real - principal exists and is scoped to its account. `auth` (and the `auth-provider` - slot behind it) answers *authentication*: "are you signed in, and as whom?". - This concern answers *authorization*: "given who you are, **may** you do this - to this resource?". `auth` already names "Authorization (RBAC)" and - "enforced server-side" as a requirement of its surface; this concern owns the - **model behind that requirement** and makes the per-handler permission check - reviewer-checkable. Do not restate signup/session lifecycle here; consume the - authenticated principal `auth` establishes. -- **`multi-tenancy`** owns **isolation between tenants** — that a record is even - *in the caller's tenant at all* (an *inter-tenant* guarantee). This concern - owns **intra-tenant permissions** — given a record the caller can legitimately - reach, *may they do this to it* ("may this member delete this project in - *their own* tenant?"). The two compose and are ordered: **the tenant predicate - must hold before a permission check is meaningful** — a correct permission - check on a record that belongs to the wrong tenant is still a cross-tenant - leak, so tenant scoping precedes authorization. `multi-tenancy` states this - reciprocally; do not fold tenant isolation into this concern, and do not let a - permission check stand in for the tenant predicate. -- **`security-owasp`** owns general hardening — injection, CSRF, secret - handling, input validation, TLS. **Broken Access Control is the OWASP umbrella - failure; this concern is the access-control *model* that prevents it.** - `security-owasp` says "authorization checked on every protected endpoint"; - this concern owns *how the decision is modeled and enforced* (deny-by-default, - a chosen model, a central decision point) so that the umbrella requirement has - a concrete, reviewer-checkable shape. - -This concern owns the one thing those do not state: **there is a deliberately -chosen permission model, every state-changing and data-returning handler -authorizes the principal against it deny-by-default, and a negative test proves -an unauthorized principal is refused.** + +For the family ownership table (auth / authorization-model / multi-tenancy / +security-owasp, plus the admin-console and unity-catalog neighbors), and the +ordering invariants (tenant predicate precedes permission; authn precedes +authz; hardening does not substitute for the model), see +[README-auth-family.md](../README-auth-family.md). + +This concern owns the one thing the rest of the family does not state: **there +is a deliberately chosen permission model, every state-changing and +data-returning handler authorizes the principal against it deny-by-default, and +a negative test proves an unauthorized principal is refused.** Broken Access +Control is the OWASP umbrella `security-owasp` names; this concern is the +access-control *model* that prevents it. ## Components diff --git a/workflows/concerns/authorization-model/practices.md b/workflows/concerns/authorization-model/practices.md index e1f322f2..73a39db8 100644 --- a/workflows/concerns/authorization-model/practices.md +++ b/workflows/concerns/authorization-model/practices.md @@ -2,13 +2,12 @@ These practices govern the **permission model and its enforcement** — *what* an authenticated principal may do, and the discipline that every privileged handler -asks and refuses by default. They sit alongside `auth` (who you are), -`multi-tenancy` (is the record even in your tenant — the predicate that precedes -any permission check), and `security-owasp` (Broken Access Control is the OWASP -umbrella this model prevents). Their one job is to make the **missing / -mis-placed authorization check** unreachable and **reviewer-checkable**. Each -MUST/SHOULD below is written so a reviewer can confirm or refute it against the -diff and the running system. +asks and refuses by default. For the family ownership table (auth / +authorization-model / multi-tenancy / security-owasp) see +[README-auth-family.md](../README-auth-family.md). Their one job is to make the +**missing / mis-placed authorization check** unreachable and +**reviewer-checkable**. Each MUST/SHOULD below is written so a reviewer can +confirm or refute it against the diff and the running system. ## Choose and record the model (Frame / Design) @@ -84,17 +83,6 @@ diff and the running system. (succeeds) and the denied (refused) cases **SHOULD** be exercised, so the guard branch is shown to exist and not be vacuously open. -## Boundary with neighbors - -- For **who the principal is** (signup, login, sessions, account bootstrap) - defer to `auth`; do not restate it here. This concern consumes the principal - and decides what it may do. -- For **is the record even in the caller's tenant** defer to `multi-tenancy`; - that predicate precedes — and never replaces — the permission check here. -- For general hardening (injection, CSRF, secrets, TLS) defer to - `security-owasp`; missing/broken authorization is its Broken Access Control - case, given a concrete model and per-handler discipline here. - ## Quality Gates - Permission **model recorded in an ADR** (RBAC / ABAC / ReBAC or a named diff --git a/workflows/concerns/caching-strategy/practices.md b/workflows/concerns/caching-strategy/practices.md index cc5caa80..3a345103 100644 --- a/workflows/concerns/caching-strategy/practices.md +++ b/workflows/concerns/caching-strategy/practices.md @@ -8,7 +8,7 @@ failure-handling policy a cache may participate in (`resilience`), or the schema of the store the cache fronts (`relational-data-modeling` / the `datastore` slot). A cache is always a derived, disposable copy; the origin is the truth. -## Cache only to hit a stated target +## Discover - Add a cache **only** to meet a named **performance NFR** against a **real read-heavy hot path or an expensive computation**. Record *which* path / @@ -18,7 +18,7 @@ slot). A cache is always a derived, disposable copy; the origin is the truth. - Before caching an expensive computation, confirm the cost is real (measured), not assumed; a material uncertainty is a `tech-spike`, not a silent cache. -## Pick the pattern from the consistency need +## Frame - Choose the read/write pattern deliberately and record **why** in the ADR: - **cache-aside (lazy)** / **read-through** for read-heavy data tolerant of @@ -31,8 +31,10 @@ slot). A cache is always a derived, disposable copy; the origin is the truth. accepted**. - Record the **consistency trade-off** the chosen pattern implies against the data's tolerance for staleness — never stumble into write-behind's window. +- Record in the ADR **what is deliberately not cached** (and why it must stay + fresh). -## State invalidation, TTL, and a staleness budget for every cache +## Design - Each cached dataset has an explicit **invalidation policy** — a TTL, explicit invalidation on write, or both — and a **named staleness budget** (the maximum @@ -41,9 +43,6 @@ slot). A cache is always a derived, disposable copy; the origin is the truth. - When using explicit invalidation, identify **every write path** that affects a cached key and invalidate/update on each; a missed write path is a stale-read bug. Record the cache **keys** and their TTLs in the technical-design. - -## Do not cache correctness-sensitive reads behind a stale copy - - Data where a stale read causes an **incorrect decision or side effect** — authorization/permission checks, balances or quotas that gate an action, anything needing **read-your-write** — is **not** served from a cache that can @@ -52,7 +51,7 @@ slot). A cache is always a derived, disposable copy; the origin is the truth. - The cache is **never the system of record**: nothing of record lives only in the cache; on a miss, eviction, or cache failure the **origin is the truth**. -## Protect hot keys from stampede +## Build - For each hot key, ensure expiry or cold start does **not** dogpile the origin: apply **single-flight / request coalescing** (one loader, others wait), @@ -62,17 +61,7 @@ slot). A cache is always a derived, disposable copy; the origin is the truth. is desired, record that as a `resilience` degradation decision — the cache composes as a fallback, it is not itself the resilience mechanism. -## Boundary with neighbors - -- The **performance NFR** owns the target; this concern owns the cache that - serves it. Do not restate the latency/throughput number — point at the NFR. -- **`resilience`** owns timeouts/retries/breakers/fallbacks and whether - stale-serve-on-failure is an accepted degradation; this concern owns the - cache's read/write/invalidation behavior. Caching ≠ resilience. -- **`relational-data-modeling`** / the **`datastore`** slot owns the system of - record the cache fronts; the cache holds a derived, disposable copy. - -## Quality Gates +## Test - Every cache **traces to a performance NFR and a real read-heavy hot path or expensive computation**; no cache exists without a target it serves (no @@ -91,3 +80,13 @@ slot). A cache is always a derived, disposable copy; the origin is the truth. TTL, or negative cache); a hot-key expiry does not dogpile the origin. - **What is deliberately not cached** (and why it must stay fresh) is recorded in the ADR. + +## Cross-cutting + +### Boundary with neighbors + +See `concern.md` for the canonical Boundary (vs the performance NFR, +`resilience`, `relational-data-modeling` / `datastore`). The cache is a +performance/staleness mechanism — defer failure-handling policy to +`resilience` and the schema of the system of record to the data-modeling / +store neighbors. diff --git a/workflows/concerns/classic-layered/practices.md b/workflows/concerns/classic-layered/practices.md index e26cff17..eedf17b8 100644 --- a/workflows/concerns/classic-layered/practices.md +++ b/workflows/concerns/classic-layered/practices.md @@ -13,22 +13,45 @@ depend directly on the data-access layer.** These practices do not require an inverted persistence boundary — if review finds the team wants one, the `architecture-style` selection is wrong and a sibling should fill the slot. -## Layering and dependency direction +## Discover + +- Confirm the product does **not** require a swappable datastore, an + isolated-testable domain, or multiple symmetric driving adapters — if it + does, the `architecture-style` is mis-selected; recommend a + dependency-inversion sibling (`onion` / `hexagonal` / `clean`). +- Per KISS/YAGNI, do NOT introduce repository interfaces, ports, or a DI + container speculatively under this concern — that ceremony belongs to a + sibling style that was deliberately selected. + +## Frame + +- If a relaxed/open-layer policy is adopted (a layer may call any lower layer), + it MUST be recorded as a deliberate decision; even then, a layer MUST NOT + call a layer **above** it. +- If the product actually needs a **swappable datastore**, an + **infrastructure-free testable domain**, or **multiple symmetric driving + adapters**, the reviewer SHOULD flag that the `architecture-style` selection + is mismatched and recommend a dependency-inversion sibling, rather than + bolting partial inversion onto a classic-layered codebase. + +## Design - The code MUST be organized into recognizable layers — **presentation → application/service → domain → data-access** — with a discoverable mapping from layer to package/module/directory. - Each layer MUST depend **only on the layer directly beneath it** (closed-layer - default). If a relaxed/open-layer policy is adopted (a layer may call any - lower layer), it MUST be recorded as a deliberate decision; even then, a layer - MUST NOT call a layer **above** it. + default). - A lower layer MUST NOT import, reference, or depend on a higher layer (verify the import graph: the data-access layer has **zero** edges to the presentation or application layers). - The presentation layer MUST NOT call the data-access layer directly or touch persistence by other means; it MUST go through the application/domain layer. +- This style does NOT require persistence interfaces to be declared in the + domain layer, and does NOT require a composition root. Reviewers MUST NOT + flag the absence of an inverted persistence boundary as a defect under this + concern — that absence is the point. -## Keep behavior in its layer +## Build - Business rules MUST live in the domain/application layer, not in presentation controllers and not in the data-access layer (no business logic in SQL, @@ -37,31 +60,7 @@ inverted persistence boundary — if review finds the team wants one, the DAOs) and contain no business decisions; the presentation layer SHOULD contain no business decisions either. -## Accept the trade-off explicitly (do not pretend to be a sibling) - -- This style does NOT require persistence interfaces to be declared in the - domain layer, and does NOT require a composition root. Reviewers MUST NOT - flag the absence of an inverted persistence boundary as a defect under this - concern — that absence is the point. -- If the product actually needs a **swappable datastore**, an - **infrastructure-free testable domain**, or **multiple symmetric driving - adapters**, the reviewer SHOULD flag that the `architecture-style` selection - is mismatched and recommend a dependency-inversion sibling, rather than - bolting partial inversion onto a classic-layered codebase. -- Per KISS/YAGNI, do NOT introduce repository interfaces, ports, or a DI - container speculatively under this concern — that ceremony belongs to a - sibling style that was deliberately selected. - -## Boundary with sibling concerns - -- The **contents** of the domain layer (aggregates, invariants, ubiquitous - language) are governed by `domain-driven-design`, not here. Verify the model - sits in the domain layer; do not restate DDD modeling rules. -- Object-level collaboration patterns inside a layer are `design-patterns-gof`; - between-system integration is `enterprise-integration-patterns`. - Classic-layered only governs the macro tier-stacking across the codebase. - -## Quality Gates +## Test - Import-graph check: dependencies point **downward only** — no layer imports a layer above it; the data-access layer has zero edges to presentation or @@ -78,3 +77,14 @@ inverted persistence boundary — if review finds the team wants one, the an isolated-testable domain, or multiple symmetric driving adapters — if it does, the `architecture-style` is mis-selected; recommend a dependency-inversion sibling (`onion` / `hexagonal` / `clean`). + +## Cross-cutting + +### Boundary with sibling concerns + +- The **contents** of the domain layer (aggregates, invariants, ubiquitous + language) are governed by `domain-driven-design`, not here. Verify the model + sits in the domain layer; do not restate DDD modeling rules. +- Object-level collaboration patterns inside a layer are `design-patterns-gof`; + between-system integration is `enterprise-integration-patterns`. + Classic-layered only governs the macro tier-stacking across the codebase. diff --git a/workflows/concerns/concurrency-model/practices.md b/workflows/concerns/concurrency-model/practices.md index 69611c4b..8c0c8280 100644 --- a/workflows/concerns/concurrency-model/practices.md +++ b/workflows/concerns/concurrency-model/practices.md @@ -108,25 +108,11 @@ reference those concerns at the seam and stay on the execution model. ## Boundary with neighbors -- **vs `enterprise-integration-patterns`**: the **channel/queue a worker - consumes** across a system boundary — its delivery guarantee, DLQ, router, - correlation id, wire shape — is EIP. The **worker pool's concurrency, the - job's idempotency, and its in-process failure visibility** are here. Idempotency - is shared vocabulary (EIP's Idempotent Receiver vs an idempotent job), split by - cause: EIP because the *channel* redelivers, this concern because the *worker* - retries. Do not restate channel/DLQ rules here. -- **vs `resilience`**: a **bounded work pool / semaphore / bounded channel that - caps in-flight work in steady state** is here (the execution model's native - bound). A **bulkhead** (per-dependency pool isolation), **load-shedding** under - overload, the **circuit breaker** and **timeout** are `resilience` (failure - containment). Backpressure as the steady-state shape of the execution model is - here; load-shedding as a failure response is resilience. Do not restate - breaker/bulkhead/shed machinery here. -- **vs `deployment-topology`**: **in-process** execution (threads / event loop / - goroutines / worker pool inside one deployable) is here; **process-level - scaling** (replica count, monolith-vs-microservices) is deployment-topology. - A horizontally-scaled fleet still needs an in-process model per replica — that - per-replica model is here. +See `concern.md` for the canonical Boundary (vs `enterprise-integration-patterns`, +`resilience`, `deployment-topology`). These practices stay on the in-process +execution model; reach to the neighbor named there for channel/DLQ rules +(EIP), bulkhead/breaker/load-shedding (`resilience`), or replica/deployable +count (`deployment-topology`). ## Quality Gates diff --git a/workflows/concerns/cqrs/practices.md b/workflows/concerns/cqrs/practices.md index b80dc228..867de661 100644 --- a/workflows/concerns/cqrs/practices.md +++ b/workflows/concerns/cqrs/practices.md @@ -11,7 +11,7 @@ aggregate, defer to DDD; when it references the event store / projections as the source of truth, defer to event sourcing. Apply these **per bounded context** that selected CQRS — not to every context in the product. -## Decide the bounded context and consistency model before applying +## Discover - Confirm the context **earns** CQRS: a **collaborative** domain (many concurrent writers), a **task-based UI** (intents map to commands), **divergent read vs. @@ -19,18 +19,25 @@ that selected CQRS — not to every context in the product. neither side. If it is simple CRUD read the same way it is written, do **NOT** apply CQRS — use a single model (record the decision; Fowler: CQRS adds risky complexity, it is difficult to use well). + +## Frame + - Choose and **record the consistency model in an ADR**: **synchronous** (read model updated in the write transaction — strongly consistent, simpler, same store) vs. **eventually consistent** (read model updated out-of-band, separate store — scalable, but a stale-read window). Name the **bounded context**, the **decisive signal**, and (if eventually consistent) the **synchronization path** and the **staleness window** consumers must tolerate. +- A **synchronous** consistency model (read model updated in the write + transaction, same store) is the simpler choice when strong read-after-write is + required; prefer it unless the asymmetry/scale signal justifies the + eventually-consistent separation. Whichever is chosen MUST match the ADR. - The selection MUST drive the artifacts: the **technical-design** SHOULD show separate command/query models, the projection/read-model, and the eventual-consistency handling; the **data-design** SHOULD show the read schema distinct from the write schema and how they stay in sync. -## Separate the write model from the read model +## Design - The model handling **commands** (state changes) MUST be **distinct** from the model serving **queries** (reads) within the CQRS-selected context — not one @@ -40,9 +47,6 @@ that selected CQRS — not to every context in the product. method that mutates *and* returns the mutated view conflates the sides. - Queries MUST read from the **read model / projection**, NOT by loading write-model aggregates to read fields off them. - -## Keep the logic in the write model; keep the read model thin - - The **write model** carries the full command-processing stack — input validation, business validation, invariants — and IS the `domain-driven-design` aggregate (defer there: a command loads the aggregate, @@ -51,9 +55,6 @@ that selected CQRS — not to every context in the product. - The **read model MUST return DTOs / view objects with no domain logic** — no invariants, no business rules, no validation stack. It is denormalized and query-shaped, optimized for retrieval. - -## Name commands for the business task - - Commands MUST be named for the **business task / intent** (`BookHotelRoom`, `RateProduct`, `SubmitOrder`), capturing user intent — NOT anemic field setters (`SetReservationStatusToReserved`). @@ -61,7 +62,7 @@ that selected CQRS — not to every context in the product. requests** in collaborative domains (a command scoped to a meaningful task conflicts less than a coarse "save everything"). -## Handle the consistency model explicitly +## Build - When the read model is **eventually consistent**, the UI/consumers MUST be designed for the **staleness window**: no read-your-write assumption across the @@ -75,30 +76,8 @@ that selected CQRS — not to every context in the product. - Projection / read-model update handlers MUST be **idempotent** — applying the same update event twice yields the same read-model state and fires any side effect at most once. -- A **synchronous** consistency model (read model updated in the write - transaction, same store) is the simpler choice when strong read-after-write is - required; prefer it unless the asymmetry/scale signal justifies the - eventually-consistent separation. Whichever is chosen MUST match the ADR. - -## Boundary with neighbors -- The **write model is the `domain-driven-design` aggregate** — defer there for - its shape, invariants, and the one-aggregate-per-transaction rule; this concern - asserts only that **queries bypass that model** and read from a projection, and - that CQRS applies **per bounded context**. -- When composed with **`event-sourcing`**, the **event store IS the write model / - source of truth and the projections ARE the read side** — the event - immutability, replay, snapshot, and projection-rebuild rules are - `event-sourcing`'s; do NOT restate them here. CQRS and event sourcing are - **independently adoptable**: CQRS needs no event log, event sourcing needs no - command/query split. -- The **read/write split is INTERNAL, not the wire contract** — a single - `api-style` surface (REST/GraphQL/gRPC/RPC) fronts the CQRS-structured service; - the command/query DTOs are not the exposed contract. Defer to `api-style` for - the wire interface; do not conflate command-vs-query (internal) with - mutation-vs-safe-verb (wire). - -## Quality Gates +## Test - CQRS is **scoped to the bounded context(s) that earn it** (collaborative / task-based UI / divergent read-write shape-or-scale / complex domain); @@ -123,3 +102,16 @@ that selected CQRS — not to every context in the product. eventual-consistency handling), and the **data-design** (read schema distinct from write schema + sync path) — verifiable that the selection is not silent drift. + +## Cross-cutting + +### Boundary with neighbors + +See `concern.md` for the canonical Boundary (vs `domain-driven-design`, +`event-sourcing`, `api-style`). Idempotent projection updates here are the +same property `enterprise-integration-patterns` requires of consumers under +at-least-once delivery — defer there for the cause, apply it here to the +read-model update path. When composed with event sourcing, the event store +becomes the write model / source of truth and the projections the read side; +the immutability/replay/rebuild rules are `event-sourcing`'s — not restated +here. diff --git a/workflows/concerns/databricks-apps/practices.md b/workflows/concerns/databricks-apps/practices.md index c8df68ce..0124741b 100644 --- a/workflows/concerns/databricks-apps/practices.md +++ b/workflows/concerns/databricks-apps/practices.md @@ -56,14 +56,11 @@ model (`unity-catalog`) — see the boundary in `concern.md`. ## Boundary with neighbors -- **vs `frontend-framework`**: the UI framework runs inside this runtime; this - concern owns hosting/identity/data wiring, not components/routing/styling. -- **vs generic `deploy-target`**: Databricks Apps **is** the deploy target here - (managed serverless) — do not stand up parallel self-hosted hosting. -- **vs `unity-catalog`**: this concern owns *that* data access flows through the - catalog and *which identity* is used; the catalog concern owns the grant model. -- **vs `security-owasp` / `auth`**: the platform supplies app service principal + - OAuth + on-behalf-of-user; compose with the app's own RBAC, do not duplicate. +See `concern.md` for the canonical Boundary (vs `frontend-framework`, generic +`deploy-target`, `unity-catalog`, `security-owasp` / `auth`). Composition in +the Databricks family: this concern hosts; `databricks-declarative-pipelines` +produces the data; `unity-catalog` governs it — each owns its piece, none +restates the others. ## Quality Gates diff --git a/workflows/concerns/databricks-declarative-pipelines/practices.md b/workflows/concerns/databricks-declarative-pipelines/practices.md index 1044ffa2..db91875f 100644 --- a/workflows/concerns/databricks-declarative-pipelines/practices.md +++ b/workflows/concerns/databricks-declarative-pipelines/practices.md @@ -54,15 +54,11 @@ cross-system messaging (`enterprise-integration-patterns`) — see the boundary ## Boundary with neighbors -- **vs domain modeling**: realize the logical model here; do not re-derive it. -- **vs `enterprise-integration-patterns`**: a pipeline reading a broker source - respects EIP's at-least-once/idempotency reality at the ingestion edge, but the - expectation/DAG/incremental machinery is **not** EIP's channel/router - machinery — keep cross-system messaging in EIP. -- **vs `unity-catalog`**: this concern produces the datasets; the catalog concern - governs them (grants/ownership/lineage). -- **vs `testing`/`verification`**: expectations are **in-pipeline** data-quality - control; compose with unit/e2e testing, do not let one substitute for the other. +See `concern.md` for the canonical Boundary (vs domain modeling, +`enterprise-integration-patterns`, `unity-catalog`, `testing` / +`verification`). Composition in the Databricks family: this concern produces +the data; `unity-catalog` governs it; `databricks-apps` hosts apps that +consume it — each owns its piece, none restates the others. ## Quality Gates diff --git a/workflows/concerns/demo-asciinema/concern.md b/workflows/concerns/demo-asciinema/concern.md index dc1a7f1e..5657663d 100644 --- a/workflows/concerns/demo-asciinema/concern.md +++ b/workflows/concerns/demo-asciinema/concern.md @@ -354,5 +354,3 @@ because they can be regenerated from scripts. Selecting this concern requires these artifacts to change (a selected concern absent from them is drift): - ADR: Asciinema scripted terminal recordings (Docker-reproducible, microsite-embedded) as the demo-reel mechanism - -## ADR References diff --git a/workflows/concerns/demo-playwright/concern.md b/workflows/concerns/demo-playwright/concern.md index cf9fbb53..367910a1 100644 --- a/workflows/concerns/demo-playwright/concern.md +++ b/workflows/concerns/demo-playwright/concern.md @@ -149,5 +149,3 @@ demonstrated. Selecting this concern requires these artifacts to change (a selected concern absent from them is drift): - ADR: Playwright reel specs (video capture, seeded data, fixed viewport) as the demo-reel mechanism - -## ADR References diff --git a/workflows/concerns/design-patterns-gof/concern.md b/workflows/concerns/design-patterns-gof/concern.md index c4a683d0..f9d80958 100644 --- a/workflows/concerns/design-patterns-gof/concern.md +++ b/workflows/concerns/design-patterns-gof/concern.md @@ -161,5 +161,3 @@ answer and naming a pattern only adds indirection. Selecting this concern requires these artifacts to change (a selected concern absent from them is drift): - ADR: each pattern introduced against a named recurring problem + recorded intent - TD: the pattern and the variability/duplication it absorbs at the collaboration point - -## ADR References diff --git a/workflows/concerns/domain-driven-design/practices.md b/workflows/concerns/domain-driven-design/practices.md index 30a28fa0..f5066d66 100644 --- a/workflows/concerns/domain-driven-design/practices.md +++ b/workflows/concerns/domain-driven-design/practices.md @@ -7,20 +7,25 @@ object/integration mechanics that may implement a role (`design-patterns-gof`, `enterprise-integration-patterns`). When a rule below references layering, defer to onion for the *how*; this concern asserts the *what*. -## Establish the language and boundaries first +## Discover + +- Classify the area's **subdomain** as core / supporting / generic. Concentrate + modeling effort on **core**; do not deeply model a **generic** subdomain that a + bought/adopted solution already solves. + +## Frame - Name the **bounded context(s)** the work lives in, and the **ubiquitous language** of each: the precise domain terms for the concepts, used identically in specs, type names, methods, and events. A term MUST NOT mean two different things inside one context. -- Classify the area's **subdomain** as core / supporting / generic. Concentrate - modeling effort on **core**; do not deeply model a **generic** subdomain that a - bought/adopted solution already solves. - When this context consumes a foreign or legacy model, translate it through an **anti-corruption layer** — the foreign concepts MUST NOT leak into this context's model. +- Each Vernon rule is a **default**; a deliberate deviation MUST be recorded in + an ADR with its justification. -## Model the building blocks with behavior +## Design - Model concepts with **identity and lifecycle** as **entities** (equality by identity); model descriptive, replaceable concepts (money, quantity, date @@ -33,9 +38,6 @@ to onion for the *how*; this concern asserts the *what*. - Use a **factory** when constructing a complex aggregate so it is born valid; use a **domain service** (stateless) only for logic that genuinely has no home on a single entity or value object (e.g. an operation spanning two aggregates). - -## Design aggregates by their invariants - - Draw the **aggregate boundary** around exactly the true invariant — the rule that MUST stay transactionally consistent — and **no more**. The aggregate boundary IS the transactional consistency boundary. @@ -50,10 +52,8 @@ to onion for the *how*; this concern asserts the *what*. must affect another, publish a **domain event** and reconcile with **eventual consistency** outside the transaction — do not enlarge the transaction to span both roots. -- Each Vernon rule is a **default**; a deliberate deviation MUST be recorded in - an ADR with its justification. -## Keep persistence out of the domain +## Build - **Repositories return and accept aggregates** — one repository per aggregate root. ORM entities, row structs, query builders, ResultSets, and raw column @@ -62,26 +62,13 @@ to onion for the *how*; this concern asserts the *what*. - The model is expressed in the **ubiquitous language**, not in storage terms; persistence conforms to the model, not the reverse. (The dependency-inversion mechanism that makes this hold is `onion-architecture`'s practice.) - -## Enforce invariants in the domain layer - - Every true invariant is checked **inside the aggregate**, in the domain layer — **not** in a controller, request handler, route, or application service, and **not** delegated to a database constraint as the sole guard. - An application service / handler coordinates (load aggregate → call a root method → save via repository); it MUST NOT contain the business rule itself. -## Boundary with neighbors - -- For **layering** (where domain/application/infrastructure live, the dependency - rule, who defines the repository interface) defer to `onion-architecture`; do - not restate it here. -- A DDD **Factory** / **Domain Event** is a domain role; the GoF mechanic or the - messaging transport that implements it belongs to `design-patterns-gof` / - `enterprise-integration-patterns`. Name the domain role; do not specify the - mechanic here. - -## Quality Gates +## Test - Each work item's bounded context and ubiquitous language are named; domain type/method/event names match that language (no synonym drift, no term with two @@ -101,3 +88,15 @@ to onion for the *how*; this concern asserts the *what*. - Descriptive concepts with invariants (money, ranges, identifiers) are modeled as immutable value objects, not bare primitives with rules scattered at call sites. + +## Cross-cutting + +### Boundary with neighbors + +- For **layering** (where domain/application/infrastructure live, the dependency + rule, who defines the repository interface) defer to `onion-architecture`; do + not restate it here. +- A DDD **Factory** / **Domain Event** is a domain role; the GoF mechanic or the + messaging transport that implements it belongs to `design-patterns-gof` / + `enterprise-integration-patterns`. Name the domain role; do not specify the + mechanic here. diff --git a/workflows/concerns/e2e-kind/concern.md b/workflows/concerns/e2e-kind/concern.md index 6bec12d0..badb7fc8 100644 --- a/workflows/concerns/e2e-kind/concern.md +++ b/workflows/concerns/e2e-kind/concern.md @@ -212,5 +212,3 @@ against the real service stack. Particularly valuable when: Selecting this concern requires these artifacts to change (a selected concern absent from them is drift): - TEST_PLAN: E2E against a real ephemeral kind cluster — all dependencies deployed, deterministic seed, self-contained lifecycle - -## ADR References diff --git a/workflows/concerns/e2e-playwright/concern.md b/workflows/concerns/e2e-playwright/concern.md index 08939d9a..59fcdd6a 100644 --- a/workflows/concerns/e2e-playwright/concern.md +++ b/workflows/concerns/e2e-playwright/concern.md @@ -15,9 +15,6 @@ e2e-framework - **Browser engines**: Chromium (default), optionally Firefox and WebKit - **Screenshot snapshots**: Visual regression detection via `toHaveScreenshot()` - **Video recording**: Full test execution videos for human review -- **Demo reel**: A dedicated slow-paced Playwright script that produces a - polished walkthrough video of the application for stakeholder review, - README embedding, or onboarding - **Trace files**: Playwright traces for debugging failures - **Test data**: Fake/seed data that exercises every meaningful UI state @@ -134,43 +131,12 @@ Playwright generates: Both directories should be gitignored. In CI, upload as artifacts. -## Demo Reel +## Demo reels -Every project with a web UI should include a Playwright demo reel script that -produces a polished walkthrough video. This replaces manual screen recordings -with a reproducible, scriptable demo. - -### Demo reel structure - -The demo reel is a single Playwright test file (e.g., `e2e/demo.spec.ts`) that: - -1. Seeds the application with comprehensive, realistic data -2. Navigates every major page and workflow at a human-readable pace -3. Uses `page.waitForTimeout()` between actions for viewing comfort -4. Produces a `.webm` video in `test-results/` that can be converted to - `.mp4` or `.gif` for embedding - -### Demo reel conventions - -- File: `e2e/demo.spec.ts` (separate from test specs) -- Viewport: 1280x720 (HD, suitable for embedding) -- Pacing: 1-2 second pauses between navigations, 2-3 seconds on key screens -- Use `test.step()` for structured logging of what each section shows -- Narrative structure: Overview (dashboard) -> Primary workflows -> Detail views -- Run with: `npx playwright test e2e/demo.spec.ts` -- Output: `test-results/demo-*/video.webm` -- The demo must run against seeded data — never against an empty state -- Re-record the demo after major UI changes - -### Converting output - -```bash -# WebM to MP4 (for README/docs) -ffmpeg -i test-results/demo-*/video.webm -c:v libx264 -preset slow -crf 22 demo.mp4 - -# WebM to GIF (for README, keep under 10MB) -ffmpeg -i test-results/demo-*/video.webm -vf "fps=10,scale=960:-1" -loop 0 demo.gif -``` +Scripted walkthrough videos are owned by **`demo-playwright`** (viewport, +pacing, narrative structure, post-processing). When a project needs a reel, +select `demo-playwright` alongside this concern; do not extend the test suite +here to double as demo capture. ## When to use @@ -185,5 +151,3 @@ Any project with a web UI that users interact with. This includes: Selecting this concern requires these artifacts to change (a selected concern absent from them is drift): - ADR: Playwright as the e2e-framework slot (real browsers, screenshots, video, traces) - TEST_PLAN: per-page + per-workflow E2E in real browsers, seeded data across UI states, committed screenshot baselines - -## ADR References diff --git a/workflows/concerns/e2e-playwright/practices.md b/workflows/concerns/e2e-playwright/practices.md index b5560ee0..583d27ec 100644 --- a/workflows/concerns/e2e-playwright/practices.md +++ b/workflows/concerns/e2e-playwright/practices.md @@ -102,18 +102,11 @@ assertion until the cue exists. - Debugging flaky tests (video shows timing/race conditions) - In CI, upload `test-results/` as a build artifact for post-hoc review -## Demo Reel -- Create `e2e/demo.spec.ts` — a single test that walks through the entire app -- Seed with realistic data that makes the demo compelling (populated states, - not empty states) -- Set viewport to 1280x720 for clean video output -- Pace with `page.waitForTimeout()`: 1-2s between clicks, 2-3s on key screens -- Use `test.step()` to narrate each section in the test output -- Structure: Dashboard overview -> key workflows -> detail pages -> settings -- Run separately: `npx playwright test e2e/demo.spec.ts` -- Output video lives in `test-results/` — convert to `.mp4` for embedding -- Re-record after significant UI changes -- The demo is documentation — keep it passing, keep it current +## Demo reels — delegated to `demo-playwright` + +Scripted walkthrough videos (viewport, pacing, narrative structure, +post-processing) belong to `demo-playwright`. Select that concern alongside +this one when a reel is needed; do not author demo specs here. ## Quality Gates - At least one core user-flow has a browser e2e test that **runs green against @@ -127,7 +120,6 @@ assertion until the cue exists. a substitute; no screenshot assertions for this cue) - Every user-facing workflow has at least one end-to-end test - Video recording is enabled (not disabled for speed) -- Demo reel script exists and produces a watchable video - No tests skip or are marked `.only` ## CI Integration diff --git a/workflows/concerns/enterprise-application-patterns/practices.md b/workflows/concerns/enterprise-application-patterns/practices.md index aa791f59..37189022 100644 --- a/workflows/concerns/enterprise-application-patterns/practices.md +++ b/workflows/concerns/enterprise-application-patterns/practices.md @@ -9,13 +9,13 @@ PoEAA choices and the machinery they pull in. They sit beside the **domain-logic and data-source patterns matched to the actual complexity, recorded, and not over-built** (KISS/YAGNI). -## Choosing the domain-logic organization +## Discover - The domain-logic pattern MUST be chosen against the **assessed complexity of - the business logic**, and the choice recorded in an ADR: **Transaction Script** - for simple, mostly independent operations; **Domain Model** when the rules, - cases, and interactions are genuinely complex; **Table Module** only when a - record-set–centric stack/UI justifies it. + the business logic**: **Transaction Script** for simple, mostly independent + operations; **Domain Model** when the rules, cases, and interactions are + genuinely complex; **Table Module** only when a record-set–centric stack/UI + justifies it. - You SHOULD start with the **simplest** organization the logic warrants and refactor *toward* a Domain Model when complexity actually arrives — not stand up a Domain Model speculatively for a thin surface. @@ -23,9 +23,13 @@ recorded, and not over-built** (KISS/YAGNI). API, batch, integration) need the same operations or a single transaction/orchestration seam is required — not as a reflexive passthrough over a single caller. +- The **session-state** placement (client / server / database) SHOULD be a + recorded choice driven by state size, statelessness/affinity needs, and + survive-restart requirements — not an accident of the framework. -## Choosing the data-source pattern (Active Record vs Data Mapper) +## Frame +- The chosen domain-logic organization MUST be recorded in an ADR. - The **Active Record vs Data Mapper** decision MUST be made explicitly and **recorded in an ADR** — it MUST NOT be left as an implicit ORM default. - Choose **Active Record** when domain logic is **simple and maps closely to the @@ -36,10 +40,19 @@ recorded, and not over-built** (KISS/YAGNI). data-source pattern** — it is the mechanism that keeps the domain model ignorant of the database and lets repositories return aggregates in domain terms. Choosing Active Record under DDD MUST be justified in the ADR. + +## Design + - A rich Domain Model MUST NOT be placed on Active Record such that business rules bend to the table shape or SQL/row concerns leak into domain objects. +- The **technical-design** reflects the chosen data-source layer + domain-logic + organization, and the **data-design** reflects how that pattern maps objects to + the schema. +- **Remote Facade** and **Data Transfer Object** MUST be introduced only at a real + **process/network boundary**; they MUST NOT be added inside a single process + where fine-grained calls are free. -## Supplying the Data Mapper machinery +## Build - When a Data Mapper backs a non-trivial Domain Model, **Unit of Work** (one coordinated commit + concurrency resolution) and **Identity Map** (one in-memory @@ -48,27 +61,7 @@ recorded, and not over-built** (KISS/YAGNI). - **Lazy Load** SHOULD be used where eager-loading a whole graph is wasteful, with attention to N+1 read patterns. -## Applying distribution and session patterns only where they pay - -- **Remote Facade** and **Data Transfer Object** MUST be introduced only at a real - **process/network boundary**; they MUST NOT be added inside a single process - where fine-grained calls are free. -- The **session-state** placement (client / server / database) SHOULD be a - recorded choice driven by state size, statelessness/affinity needs, and - survive-restart requirements — not an accident of the framework. - -## Staying in your lane - -- When a construct carries **business meaning** (an aggregate, an invariant, a - ubiquitous-language name), the domain semantics belong to - `domain-driven-design`; PoEAA describes only the persistence/orchestration - **mechanics**. For **Repository** and **Service Layer**, name the domain meaning - per DDD and let this concern own the persistence/orchestration machinery. -- The patterns here fill layers; **which layer code lives in and the dependency - direction** belong to `onion-architecture` and MUST NOT be restated. A **generic - intra-process OO collaboration** is `design-patterns-gof`, not a PoEAA pattern. - -## Quality Gates +## Test - The **domain-logic organization** (Transaction Script vs Domain Model vs Table Module) is recorded in an ADR and matches the assessed complexity — no Domain @@ -91,3 +84,16 @@ recorded, and not over-built** (KISS/YAGNI). - Every pattern present is named to match the mechanic actually implemented (no one-method "Service Layer", no row-returning "Repository") and routes domain meaning, macro layering, and generic OO mechanics to their owning concerns. + +## Cross-cutting + +### Staying in your lane + +- When a construct carries **business meaning** (an aggregate, an invariant, a + ubiquitous-language name), the domain semantics belong to + `domain-driven-design`; PoEAA describes only the persistence/orchestration + **mechanics**. For **Repository** and **Service Layer**, name the domain meaning + per DDD and let this concern own the persistence/orchestration machinery. +- The patterns here fill layers; **which layer code lives in and the dependency + direction** belong to `onion-architecture` and MUST NOT be restated. A **generic + intra-process OO collaboration** is `design-patterns-gof`, not a PoEAA pattern. diff --git a/workflows/concerns/enterprise-integration-patterns/practices.md b/workflows/concerns/enterprise-integration-patterns/practices.md index 5ebef276..c6821b5a 100644 --- a/workflows/concerns/enterprise-integration-patterns/practices.md +++ b/workflows/concerns/enterprise-integration-patterns/practices.md @@ -91,19 +91,13 @@ they reference those concerns at the seam and stay on the wire. ## Boundary with neighbors -- **vs `domain-driven-design`**: DDD decides the **fact and its invariant** — a - domain event reconciling aggregates inside a bounded context. EIP decides the - **channel, delivery guarantee, and wire shape** that carries that fact across - the boundary. Do not re-decide the domain model here; do not let DDD own the - transport. -- **vs `onion-architecture`**: messaging clients, gateways, and consumers are - **outer-ring** adapters; the inner layer declares the port, the EIP adapter - implements it. Do not restate the dependency rule; do honor it (no broker - import inward). -- **vs `design-patterns-gof`**: GoF Observer / an in-process event bus is - **not** a Message Channel substitute for a real system boundary — it has no - durability, delivery guarantee, retry, or dead-letter path. Use a channel when - the boundary is real. +See `concern.md` for the canonical Boundary (vs `domain-driven-design`, +`onion-architecture`, `design-patterns-gof`). EIP owns the at-least-once +channel-delivery model that *causes* the redelivery downstream consumers must +absorb — `concurrency-model` (in-process workers), `event-sourcing` (event +handlers), and `cqrs` (projection updaters) each apply the resulting +idempotency requirement at their own surface; this concern does not restate +their application of it. ## Quality Gates diff --git a/workflows/concerns/event-sourcing/practices.md b/workflows/concerns/event-sourcing/practices.md index 503b7c37..21c2b506 100644 --- a/workflows/concerns/event-sourcing/practices.md +++ b/workflows/concerns/event-sourcing/practices.md @@ -96,20 +96,14 @@ not to every entity in the product. ## Boundary with neighbors -- The **aggregate, its invariants, and its emitted domain events** are - `domain-driven-design`'s — defer there for the model; this concern persists - and rehydrates that aggregate. -- The **event store is an outer-ring adapter** behind an inner-declared port — - defer to `onion-architecture` for the layering/dependency direction; do not - restate the dependency rule. -- A **stored event is not an integration message**. Publishing events to other - systems/contexts (channels, transformation, delivery) is - `enterprise-integration-patterns`' — an event handler may emit an integration - event, but do not conflate the stream-of-record with the wire message. -- The **read side / projections** are commonly the **CQRS** read model; name - CQRS as the companion, do not define the full command/query segregation here. -- Use a **real event store** (per-entity stream reads + optimistic concurrency), - not a message broker, as the system of record. +See `concern.md` for the canonical Boundary (vs `domain-driven-design`, +`enterprise-integration-patterns`, `onion-architecture`, `sample-data`, and +the **CQRS** companion). Idempotent application of events here is the same +property `enterprise-integration-patterns` requires of consumers under +at-least-once delivery — defer there for the channel-delivery rule, apply it +here to projection/read-model updates. The event store is a real event store +(per-entity stream reads + optimistic concurrency), not a message broker +standing in as the system of record. ## Quality Gates diff --git a/workflows/concerns/go-std/concern.md b/workflows/concerns/go-std/concern.md index a3f921b4..a0c22dbb 100644 --- a/workflows/concerns/go-std/concern.md +++ b/workflows/concerns/go-std/concern.md @@ -58,5 +58,3 @@ All Go projects — CLIs, services, libraries. The standard toolchain and Selecting this concern requires these artifacts to change (a selected concern absent from them is drift): - ADR: Go + standard toolchain (gofmt, golangci-lint, gosec, govulncheck) as the language-runtime - TD: error-wrapping, context-passing, interface-in-consumer conventions; lint baseline - -## ADR References diff --git a/workflows/concerns/hexagonal-architecture/practices.md b/workflows/concerns/hexagonal-architecture/practices.md index 80fdb9b9..3c584e33 100644 --- a/workflows/concerns/hexagonal-architecture/practices.md +++ b/workflows/concerns/hexagonal-architecture/practices.md @@ -15,7 +15,18 @@ The differentiator versus the sibling dependency-inversion styles (`onion` / ports** and the **plurality of adapters per port**, not on an internal concentric ring layout. -## Ports owned by the core; dependencies point inward +## Discover + +- Apply full ports-and-adapters only when the **selection signals** in + `concern.md` hold — **multiple symmetric driving and/or driven adapters** + around one core. For a thin CRUD app with one UI and one datastore and no + prospect of a second adapter, the port ceremony is over-engineering — prefer + `classic-layered`, recorded as the `architecture-style` choice. +- Per KISS/YAGNI, do NOT introduce a port for a boundary that has exactly one + adapter and no realistic prospect of a second, **unless** the port is needed + to keep the core driveable/testable in isolation. + +## Design - Code MUST be organized into an **application core** plus **adapters**, with a discoverable mapping from core / driving-adapter / driven-adapter to @@ -27,9 +38,6 @@ concentric ring layout. - A **driven (secondary) port** — the interface for persistence or an external system — MUST be declared **in the core** and implemented in a secondary adapter. The port interface MUST NOT live in the adapter's package. - -## Symmetry of driving and driven sides - - **Driving (primary) adapters** (HTTP, CLI, queue consumer, GUI, test harness) MUST call the core only through **driving ports**. **Driven (secondary) adapters** (SQL/ORM, message bus, email, third-party clients) MUST be invoked @@ -41,8 +49,12 @@ concentric ring layout. attach a second adapter (a test harness on a driving port; an in-memory fake on a driven port) without modifying the core. A port whose single implementation is hard-wired is a defect. +- Data crossing a port MUST be expressed in the **core's own terms** (core + objects / DTOs the core owns). Transport shapes (HTTP request/response + objects, ORM rows) MUST be translated **inside the adapter** and MUST NOT + cross the port into the core. -## Configurable dependency at the boundary +## Build - Concrete adapters MUST be instantiated and wired to the core's ports **at configuration time by the composition root** — the only place that names @@ -50,40 +62,12 @@ concentric ring layout. adapter. - Swapping an adapter (different datastore, CLI instead of HTTP) MUST require changes only in that adapter and the wiring, never in the core. -- Data crossing a port MUST be expressed in the **core's own terms** (core - objects / DTOs the core owns). Transport shapes (HTTP request/response - objects, ORM rows) MUST be translated **inside the adapter** and MUST NOT - cross the port into the core. - -## Keep the core driveable in isolation - - The core SHOULD be **exercisable through ports with no real devices or databases present** — driven by a test (primary) adapter and backed by fake (secondary) adapters. (Writing those tests is the `testing` concern; this practice only requires the port seams to exist on both sides.) -## Match the discipline to the product (avoid over-engineering) - -- Apply full ports-and-adapters only when the **selection signals** in - `concern.md` hold — **multiple symmetric driving and/or driven adapters** - around one core. For a thin CRUD app with one UI and one datastore and no - prospect of a second adapter, the port ceremony is over-engineering — prefer - `classic-layered`, recorded as the `architecture-style` choice. -- Per KISS/YAGNI, do NOT introduce a port for a boundary that has exactly one - adapter and no realistic prospect of a second, **unless** the port is needed - to keep the core driveable/testable in isolation. - -## Boundary with sibling concerns - -- The **contents** of the core are governed by `domain-driven-design`, not here. - Verify the model sits inside the hexagon behind ports; do not restate DDD - modeling rules. -- Object-level collaboration patterns inside a layer are `design-patterns-gof`; - between-system integration is `enterprise-integration-patterns` (its messaging - endpoints are driven adapters under this concern). Hexagonal governs only the - macro port boundary across the codebase. - -## Quality Gates +## Test - Import-graph check: the application core has **zero** dependency edges to any adapter, framework, transport, or driver package. @@ -102,3 +86,15 @@ concentric ring layout. - Selection-fit check: multiple symmetric driving/driven adapters justify the port ceremony; ports-and-adapters is not wrapped around a single-UI, single-datastore thin-CRUD app (else re-select `classic-layered`). + +## Cross-cutting + +### Boundary with sibling concerns + +- The **contents** of the core are governed by `domain-driven-design`, not here. + Verify the model sits inside the hexagon behind ports; do not restate DDD + modeling rules. +- Object-level collaboration patterns inside a layer are `design-patterns-gof`; + between-system integration is `enterprise-integration-patterns` (its messaging + endpoints are driven adapters under this concern). Hexagonal governs only the + macro port boundary across the codebase. diff --git a/workflows/concerns/hugo-hextra/concern.md b/workflows/concerns/hugo-hextra/concern.md index 891379c6..0a229a72 100644 --- a/workflows/concerns/hugo-hextra/concern.md +++ b/workflows/concerns/hugo-hextra/concern.md @@ -230,5 +230,3 @@ minimal configuration. Selecting this concern requires these artifacts to change (a selected concern absent from them is drift): - ADR: Hugo (extended) + Hextra theme via Hugo Modules, GitHub Pages deploy, as the microsite stack - TD: website/ layout, pinned theme/Hugo versions, Hugo config (enableGitInfo, unsafe HTML) - -## ADR References diff --git a/workflows/concerns/i18n-icu/concern.md b/workflows/concerns/i18n-icu/concern.md index c8c84700..7ab16c96 100644 --- a/workflows/concerns/i18n-icu/concern.md +++ b/workflows/concerns/i18n-icu/concern.md @@ -30,5 +30,3 @@ far cheaper than retrofitting. Selecting this concern requires these artifacts to change (a selected concern absent from them is drift): - TD: ICU MessageFormat, externalized string catalogs, locale-aware formatting, declared default locale - DESIGN_SYSTEM: bidirectional (LTR/RTL) text support - -## ADR References diff --git a/workflows/concerns/k8s-kind/concern.md b/workflows/concerns/k8s-kind/concern.md index 1e562bf9..973b266d 100644 --- a/workflows/concerns/k8s-kind/concern.md +++ b/workflows/concerns/k8s-kind/concern.md @@ -6,6 +6,16 @@ infrastructure ## Areas infra +## Boundary + +This concern is a **runtime implementation** that hosts whatever deployables +`deployment-topology` chose to ship (one modular monolith or many +microservices — both run here). It owns cluster / Helm chart / image-build / +local-kind workflow mechanics. It does **not** decide the deployable count +or seams (`deployment-topology`), the per-process operational contract each +deployable honors (`twelve-factor`), or the telemetry the deployables emit +(`o11y-otel`). + ## Components - **Local cluster**: `kind` (Kubernetes in Docker) — NOT docker-compose @@ -34,5 +44,3 @@ services need service discovery, ingress, or multi-container orchestration. Selecting this concern requires these artifacts to change (a selected concern absent from them is drift): - ADR: Kubernetes + kind (Helm, reproducible images) as deployment topology — not docker-compose - IMPLEMENTATION_PLAN: Helm charts + env values files, image-build/tag, kind create + install + port-forward workflow - -## ADR References diff --git a/workflows/concerns/mcp-server/practices.md b/workflows/concerns/mcp-server/practices.md index 5ce5b9a0..ac6dff1a 100644 --- a/workflows/concerns/mcp-server/practices.md +++ b/workflows/concerns/mcp-server/practices.md @@ -11,11 +11,14 @@ styles), re-specify the general threat model (`security-owasp`), or govern asynchronous messaging (`enterprise-integration-patterns`); they reference those at the seam and stay on the MCP surface. -## Decide MCP is the right surface, then model the primitives +## Discover - Confirm the **consumer is an LLM agent / AI client** consuming tools, data, or prompts for autonomous use (the `api-style` selection signal for MCP). If the consumer is a human or another service, the surface is REST/gRPC/etc., not MCP. + +## Frame + - Map each capability to the **right primitive by control surface**: - **Tool** — a **model-invoked action with effects** (write, send, call an external API); one operation, **typed JSON-Schema input + output**. @@ -27,7 +30,7 @@ at the seam and stay on the MCP surface. passive data through a tool the model must decide to call. Record the exposed primitives in an ADR. -## Write each tool description as the model's contract +## Design - Give every tool a **precise, honest, scoped description** stating what it does, when to use it, and what it does NOT do — the model selects tools on this text @@ -38,9 +41,6 @@ at the seam and stay on the MCP surface. - Do **not** embed steering instructions in a description beyond describing the tool, and treat **consumed** third-party tool descriptions as untrusted (tool poisoning, below). - -## Gate destructive tools behind human-in-the-loop - - A tool with **irreversible or high-impact effect** (delete, send, pay, deploy, mutate external state) **requires explicit user confirmation at invocation** — a consent the user sees *before* the effect happens. Read-only/safe tools may @@ -48,8 +48,12 @@ at the seam and stay on the MCP surface. decision. - Surface tool execution (approval dialog and/or activity log) so the user can **see what a tool does before authorizing it**. +- Use **stdio** for a **local** server (runs as a subprocess, JSON-RPC over + stdin/stdout, credentials from the environment) — preferred where the client + runs the server locally. Use **Streamable HTTP** for a **remote / multi-client** + server (single MCP endpoint, POST/GET, optional SSE streaming). -## Validate every tool input; scope every tool least-privilege +## Build - **Validate and coerce each tool's arguments against its JSON Schema at the boundary** before acting — the caller is an LLM that can emit malformed, @@ -59,9 +63,6 @@ at the seam and stay on the MCP surface. `full-access` / omnibus scopes, no bundling unrelated privileges up front. Prefer **progressive scope elevation** (start minimal, elevate on first privileged use) so a stolen token's blast radius stays small. - -## Defend against tool poisoning and rug-pulls (when consuming/aggregating servers) - - Treat the **descriptions and schemas of tools this product consumes** as a prompt-injection / supply-chain surface: unless the source server is trusted, do not let consumed tool metadata steer the agent. Pin/verify metadata from @@ -69,20 +70,6 @@ at the seam and stay on the MCP surface. - Detect **changed tool definitions** and **re-consent** rather than silently honoring a tool whose behavior/description changed after the user approved an earlier form (rug-pull). - -## Choose the transport and lock it down - -- Use **stdio** for a **local** server (runs as a subprocess, JSON-RPC over - stdin/stdout, credentials from the environment) — preferred where the client - runs the server locally. Use **Streamable HTTP** for a **remote / multi-client** - server (single MCP endpoint, POST/GET, optional SSE streaming). -- For a local HTTP server, **bind to localhost** (not `0.0.0.0`) and **validate - the `Origin` header** to prevent DNS-rebinding. Restrict a local HTTP server's - access (auth token, or IPC/unix socket) so other local processes cannot drive - it. - -## Authorize the HTTP transport with OAuth 2.1 — validate audience, never pass through - - An **HTTP-transport** MCP server is an **OAuth 2.1 resource server**: publish protected-resource metadata (RFC 9728), require `Authorization: Bearer` on every request, and **validate that each token was issued specifically for this @@ -100,25 +87,14 @@ at the seam and stay on the MCP surface. random, **user-bound** session IDs (`:`) — never a session ID as proof of identity, never predictable session IDs. -## Boundary with neighbors +## Deploy -- **vs `api-style`**: api-style selects the interface style and *places MCP* - among them (the agent-facing option) and records the selection; this concern - holds the MCP-specific discipline once MCP is chosen. Do not re-run the - style-selection guide here. -- **vs `security-owasp`**: security-owasp owns the baseline threat model (OWASP - Top 10, injection, access control, secrets, TLS); this concern adds the - **agent-exposure** layer (tool poisoning, rug-pull, confused-deputy, token - passthrough, over-broad tool scopes). Compose; do not restate the Top 10. -- **vs `enterprise-integration-patterns`**: MCP is a **synchronous** tool/ - resource interface the agent calls and awaits; a tool that *enqueues* work - writes to an EIP channel — the `tools/call` is this concern, the channel is - EIP's. Do not model MCP as an async message bus. -- **vs `auth`/`auth-local-sessions`**: those own the product's own user login; - this concern owns the **OAuth 2.1 agent-token boundary** for the MCP HTTP - transport (audience binding, no passthrough). Compose; do not conflate. +- For a local HTTP server, **bind to localhost** (not `0.0.0.0`) and **validate + the `Origin` header** to prevent DNS-rebinding. Restrict a local HTTP server's + access (auth token, or IPC/unix socket) so other local processes cannot drive + it. -## Quality Gates +## Test - **Each capability is modeled to the right primitive** — model-invoked actions with effects are **tools** (typed JSON-Schema input/output, single-purpose), @@ -144,3 +120,23 @@ at the seam and stay on the MCP surface. - **Consumed/aggregated third-party tool metadata is treated as untrusted** — defense against tool poisoning, and re-consent on changed tool definitions (rug-pull) where the product proxies other MCP servers. + +## Cross-cutting + +### Boundary with neighbors + +- **vs `api-style`**: api-style selects the interface style and *places MCP* + among them (the agent-facing option) and records the selection; this concern + holds the MCP-specific discipline once MCP is chosen. Do not re-run the + style-selection guide here. +- **vs `security-owasp`**: security-owasp owns the baseline threat model (OWASP + Top 10, injection, access control, secrets, TLS); this concern adds the + **agent-exposure** layer (tool poisoning, rug-pull, confused-deputy, token + passthrough, over-broad tool scopes). Compose; do not restate the Top 10. +- **vs `enterprise-integration-patterns`**: MCP is a **synchronous** tool/ + resource interface the agent calls and awaits; a tool that *enqueues* work + writes to an EIP channel — the `tools/call` is this concern, the channel is + EIP's. Do not model MCP as an async message bus. +- **vs `auth`/`auth-local-sessions`**: those own the product's own user login; + this concern owns the **OAuth 2.1 agent-token boundary** for the MCP HTTP + transport (audience binding, no passthrough). Compose; do not conflate. diff --git a/workflows/concerns/multi-tenancy/concern.md b/workflows/concerns/multi-tenancy/concern.md index c0769930..ba96ef1f 100644 --- a/workflows/concerns/multi-tenancy/concern.md +++ b/workflows/concerns/multi-tenancy/concern.md @@ -21,38 +21,19 @@ data, api This concern owns **isolation between tenants** — that tenant A can never read or write tenant B's data, on any path, and the architecture that delivers that guarantee (which resources are shared vs dedicated). It is composable and does -**not** fill a slot. Three neighbours stay distinct: - -- **`auth`** owns **who you are** — signup, login, sessions, and that a real - principal exists and is scoped to its account. `auth` establishes the - authenticated principal and the account/tenant it belongs to; **multi-tenancy - consumes that principal** and enforces that *every data access stays inside - that tenant's boundary*. Auth answers "are you signed in as someone in - tenant T?"; multi-tenancy answers "given that, can any code path reach data - outside T?". `auth` already states isolation-through-the-principal as a - requirement; this concern owns the **enforcement and isolation model** behind - it and makes the cross-tenant guard reviewer-checkable. Do not restate - signup/session lifecycle here. -- **`authorization-model`** (a sibling concern) owns **what a user may do - *within* a tenant** — roles, permissions, and the actions a principal is - allowed to perform on resources they can already legitimately reach. - Authorization is *intra-tenant* ("may this member delete this project in - *their own* tenant?"). Multi-tenancy is *inter-tenant* ("is this project even - in the caller's tenant at all?"). A correct authorization check on a record - that belongs to the wrong tenant is still a cross-tenant leak — the tenant - predicate must hold *before* the permission check is even meaningful. Compose - the two; do not fold authorization rules into this concern. -- **`security-owasp`** owns general hardening — injection, CSRF, secret - handling, input validation, TLS. Cross-tenant access is an instance of OWASP - **Broken Access Control / IDOR**; this concern is the tenant-specific, - reviewer-checkable refinement of that hardening. `security-owasp` says - "authorization checked on every protected endpoint"; multi-tenancy says - "every data access carries the tenant predicate, derived from the principal". - -This concern owns the one thing those do not state: **there is no query path -that resolves a record without the tenant predicate, and tenant identity is -derived from the authenticated principal — never from a client-supplied id -alone.** +**not** fill a slot. + +For the family ownership table (auth / authorization-model / multi-tenancy / +security-owasp, plus the admin-console and unity-catalog neighbors), and the +ordering invariants (tenant predicate precedes permission; authn precedes +authz), see [README-auth-family.md](../README-auth-family.md). + +This concern owns the one thing the rest of the family does not state: **there +is no query path that resolves a record without the tenant predicate, and +tenant identity is derived from the authenticated principal — never from a +client-supplied id alone.** Cross-tenant access is the OWASP Broken Access +Control / IDOR case `security-owasp` names at the umbrella level; this concern +is the tenant-specific, reviewer-checkable refinement. ## Components diff --git a/workflows/concerns/multi-tenancy/practices.md b/workflows/concerns/multi-tenancy/practices.md index 18f0d9d3..7a9f08e8 100644 --- a/workflows/concerns/multi-tenancy/practices.md +++ b/workflows/concerns/multi-tenancy/practices.md @@ -1,12 +1,12 @@ # Practices: Multi-Tenancy These practices govern **isolation between tenants** — that no code path lets -one tenant reach another's data. They sit alongside `auth` (who you are), -`authorization-model` (what you may do *within* a tenant), and `security-owasp` -(general hardening; cross-tenant access is Broken Access Control / IDOR). Their -one job is to make the **cross-tenant leak** unreachable and **reviewer-checkable**. -Each MUST/SHOULD below is written so a reviewer can confirm or refute it against -the diff and the running system. +one tenant reach another's data. For the family ownership table (auth / +authorization-model / multi-tenancy / security-owasp) see +[README-auth-family.md](../README-auth-family.md). Their one job is to make the +**cross-tenant leak** unreachable and **reviewer-checkable**. Each MUST/SHOULD +below is written so a reviewer can confirm or refute it against the diff and +the running system. ## Choose and record the isolation model (Frame / Design) @@ -92,19 +92,6 @@ the diff and the running system. - Offboarding **MUST** be able to export and delete a single tenant's data without affecting other tenants. -## Boundary with neighbors - -- For **who the principal is** (signup, login, sessions, account bootstrap) - defer to `auth`; do not restate it here. This concern consumes the principal. -- For **what a user may do within their tenant** (roles, permissions) defer to - `authorization-model`; this concern only guarantees the record is in the - caller's tenant in the first place. -- For general hardening (injection, CSRF, secrets, TLS, parameterized queries) - defer to `security-owasp`; cross-tenant access is its Broken Access Control / - IDOR case, refined here for tenants. -- For the tenant as part of the domain model (tenant as aggregate identity, - invariants scoped to a tenant) compose with `domain-driven-design`. - ## Quality Gates - Isolation model recorded in an ADR with its trade-off justification — not diff --git a/workflows/concerns/o11y-otel/concern.md b/workflows/concerns/o11y-otel/concern.md index 2b351519..343cc70b 100644 --- a/workflows/concerns/o11y-otel/concern.md +++ b/workflows/concerns/o11y-otel/concern.md @@ -30,5 +30,3 @@ for production debugging, performance monitoring, and incident response. Selecting this concern requires these artifacts to change (a selected concern absent from them is drift): - ADR: OpenTelemetry (traces/metrics/logs) as the observability standard - TD: RED metrics on endpoints, trace-context propagation, structured JSON logs with correlation IDs - -## ADR References diff --git a/workflows/concerns/onion-architecture/practices.md b/workflows/concerns/onion-architecture/practices.md index 742b76f5..aeb896db 100644 --- a/workflows/concerns/onion-architecture/practices.md +++ b/workflows/concerns/onion-architecture/practices.md @@ -8,7 +8,18 @@ object-level patterns (`design-patterns-gof`), not between-system messaging (`enterprise-integration-patterns`). Where DDD is also selected, its domain model is exactly what lives in the core ring described here. -## Layering and dependency direction +## Discover + +- Apply the full ring layering only when the **selection signals** in + `concern.md` hold (non-trivial domain logic, swappable infrastructure, or a + testable-domain requirement). For **thin CRUD / forms-over-data** with little + behavior, the ring ceremony is over-engineering — prefer the + `classic-layered` slot filler, recorded as the `architecture-style` choice. +- Per KISS/YAGNI, do not introduce an interface for a boundary that has exactly + one implementation and no realistic prospect of a second **unless** it is + needed to keep the domain testable in isolation. + +## Design - The code MUST be organized into concentric rings — **domain model core → domain services → application services → outer ring (infrastructure / UI / @@ -23,9 +34,6 @@ model is exactly what lives in the core ring described here. dependency still points inward** — a controller/handler depends on an application service; the application/domain layers MUST NOT depend on the controller. - -## Dependency inversion at the boundary - - Interfaces the core needs (repositories, gateways, ports, notifiers) MUST be **declared in the inner layer** (domain or application) in domain terms, and **implemented in the outer ring**. The interface and its concrete @@ -33,16 +41,16 @@ model is exactly what lives in the core ring described here. - Inner-layer code MUST depend on these interfaces, never on the concrete outer-ring class. Inner code MUST NOT `new`/construct or `import` a concrete infrastructure implementation directly. -- Concrete outer-ring implementations MUST be **injected at runtime by the - composition root** (the entrypoint / DI container / `main`). The composition - root is the **only** place that names concrete infrastructure types. - Data crossing a boundary MUST be expressed in **inner-layer terms** — domain objects or DTOs the inner layer owns. ORM rows, framework request/response objects, and other outer-ring shapes MUST NOT leak into the domain or application layers; translate at the boundary. -## Keep the core infrastructure-agnostic +## Build +- Concrete outer-ring implementations MUST be **injected at runtime by the + composition root** (the entrypoint / DI container / `main`). The composition + root is the **only** place that names concrete infrastructure types. - The domain and application layers SHOULD be **buildable/exercisable without any infrastructure present** — substituting a fake/stub adapter for each inner-layer interface should compile and run. (Writing those tests is the @@ -52,28 +60,7 @@ model is exactly what lives in the core ring described here. framework) SHOULD require changes only in the outer ring and the composition root, not in the core. -## Match the discipline to the product (avoid over-engineering) - -- Apply the full ring layering only when the **selection signals** in - `concern.md` hold (non-trivial domain logic, swappable infrastructure, or a - testable-domain requirement). For **thin CRUD / forms-over-data** with little - behavior, the ring ceremony is over-engineering — prefer the - `classic-layered` slot filler, recorded as the `architecture-style` choice. -- Per KISS/YAGNI, do not introduce an interface for a boundary that has exactly - one implementation and no realistic prospect of a second **unless** it is - needed to keep the domain testable in isolation. - -## Boundary with sibling concerns - -- The **contents** of the core (aggregates, entities, value objects, - invariants, ubiquitous language) are governed by `domain-driven-design`, not - here. Do not restate DDD modeling rules in Onion review; do verify the model - sits in the core ring with inward-only dependencies. -- Object-level collaboration patterns inside a layer are `design-patterns-gof`; - between-system integration is `enterprise-integration-patterns`. Onion only - governs the macro dependency structure across the codebase. - -## Quality Gates +## Test - Import-graph check: the domain layer has **zero** dependency edges to infrastructure / framework / ORM / web / outer-ring packages. @@ -89,3 +76,15 @@ model is exactly what lives in the core ring described here. domain or application layers — boundary translation is present. - The `architecture-style` selection fits the product: ring layering is not wrapped around a thin-CRUD app (else re-select `classic-layered`). + +## Cross-cutting + +### Boundary with sibling concerns + +- The **contents** of the core (aggregates, entities, value objects, + invariants, ubiquitous language) are governed by `domain-driven-design`, not + here. Do not restate DDD modeling rules in Onion review; do verify the model + sits in the core ring with inward-only dependencies. +- Object-level collaboration patterns inside a layer are `design-patterns-gof`; + between-system integration is `enterprise-integration-patterns`. Onion only + governs the macro dependency structure across the codebase. diff --git a/workflows/concerns/python-uv/concern.md b/workflows/concerns/python-uv/concern.md index 85f7b40c..e5583b96 100644 --- a/workflows/concerns/python-uv/concern.md +++ b/workflows/concerns/python-uv/concern.md @@ -43,5 +43,3 @@ and script running. Selecting this concern requires these artifacts to change (a selected concern absent from them is drift): - ADR: Python 3.12+ + uv (ruff, pyright, pytest) as the language-runtime - TD: pyproject.toml layout, dependency-group conventions, branch-coverage floor - -## ADR References diff --git a/workflows/concerns/react-nextjs/concern.md b/workflows/concerns/react-nextjs/concern.md index daeb4045..16982549 100644 --- a/workflows/concerns/react-nextjs/concern.md +++ b/workflows/concerns/react-nextjs/concern.md @@ -13,7 +13,9 @@ frontend-framework - **UI Framework**: React 19 — functional components and hooks only - **Meta-framework**: Next.js 15 — App Router (not Pages Router) -- **Component library**: shadcn/ui (copied components, not npm dependency) + Radix UI primitives +- **UI primitives / component library**: see `ux-radix` concern (canonical + owner of the Radix + shadcn/ui prescription) — this concern does NOT + prescribe a component library directly - **Styling**: Tailwind CSS 4 — utility-first, no CSS-in-JS - **Forms**: react-hook-form + @hookform/resolvers with Zod schemas - **Data tables**: TanStack React Table 8 — headless, sortable, filterable, virtualizable @@ -28,7 +30,9 @@ frontend-framework - No `React.FC` type annotation — use plain function signatures with typed props - Forms must use react-hook-form with uncontrolled components — no `useState` per field - Validation schemas live in the shared package and are reused on frontend and backend -- shadcn/ui components are copied into the project — do not install as npm dependency +- UI primitives and component-library choice are governed by the `ux-radix` + concern (canonical owner of shadcn/ui + Radix); compose `ux-radix` when this + concern is selected for UI work - Tailwind config extends the design system tokens (colors, spacing, typography) - E2E tests use Playwright, not Cypress or Selenium @@ -39,7 +43,8 @@ frontend-framework - `React.FC` or `React.FunctionComponent` → use plain typed functions - `useState` for every form field → must use react-hook-form - `styled-components`, `emotion`, or `css-modules` → use Tailwind utility classes -- `@shadcn/ui` as npm dependency → must be copied components +- Re-prescribing shadcn/ui or Radix primitives here → defer to `ux-radix` + (canonical owner); this concern only references that prescription - `cypress` or `selenium` → use Playwright - `getServerSideProps` or `getStaticProps` → use Server Components or route handlers - Inline `fetch` in components without error/loading states → use data fetching pattern with Suspense boundaries @@ -47,18 +52,22 @@ frontend-framework ## When to use React + Next.js frontend applications. Compose with `typescript-bun` for the -base TypeScript and Bun runtime concern. This concern adds React-specific UI -patterns, component library conventions, and E2E testing requirements. +base TypeScript and Bun runtime concern, and compose with `ux-radix` for UI +primitives and component-library prescription (shadcn/ui + Radix). This +concern adds React-specific framework patterns (App Router, Server Components, +forms, data tables) and E2E testing requirements; it does NOT prescribe a +component library. ## Artifact Impact Selecting this concern requires these artifacts to change (a selected concern absent from them is drift): -- ADR: React 19 + Next.js App Router as the frontend-framework slot; shadcn/Radix/Tailwind +- ADR: React 19 + Next.js App Router as the frontend-framework slot; Tailwind for styling (component-library prescription is owned by `ux-radix`) - TD: App Router + Server Components, react-hook-form + Zod, shared validation schemas -- DESIGN_SYSTEM: Tailwind config extends design-system tokens; shadcn/ui component conventions +- DESIGN_SYSTEM: Tailwind config extends design-system tokens (component-library conventions live with `ux-radix`) - TEST_PLAN: Playwright E2E (not Cypress/Selenium) ## ADR References - ADR-010: Frontend validation architecture (Zod shared schemas) -- ADR-011: ERP component library and navigation (shadcn/ui + Radix + Tailwind) +- ADR-011: ux-radix owns the Radix and shadcn component-library prescription; + this concern references rather than re-prescribing diff --git a/workflows/concerns/react-nextjs/practices.md b/workflows/concerns/react-nextjs/practices.md index 089fc3a6..b0cf6151 100644 --- a/workflows/concerns/react-nextjs/practices.md +++ b/workflows/concerns/react-nextjs/practices.md @@ -9,8 +9,10 @@ - Use Next.js App Router: layouts in `app/layout.tsx`, pages in `app/page.tsx`, route groups with `(groupName)/` - Default to React Server Components — add `"use client"` only for interactive components (forms, modals, dropdowns, client state) - Component hierarchy: page (server) → layout (server) → interactive widget (client) -- shadcn/ui for UI primitives — copy components via `npx shadcn@latest add `, customize in `components/ui/` -- Radix UI for accessible headless primitives underlying shadcn +- UI primitives and component library: see the `ux-radix` concern — it is the + canonical owner of the shadcn/ui + Radix prescription (copy-not-install, + customization, primitive selection). Compose `ux-radix` when this concern + is selected for UI work. - Design tokens (colors, spacing, typography) in `tailwind.config.ts` — components reference tokens, not raw values - Forms: one Zod schema per entity in `@apogee/shared`, resolved via `@hookform/resolvers/zod` - Data tables: TanStack React Table with column definitions typed against shared schemas @@ -22,7 +24,8 @@ - Hooks: `use` prefix, one file per hook in `hooks/` directory - Server Components: default export async functions, fetch data directly - Client Components: `"use client"` directive at top of file, minimize scope -- Tailwind classes: use `cn()` utility (from shadcn) for conditional classes — no `classnames` or `clsx` separately +- Tailwind classes: use a `cn()` utility for conditional classes — no + `classnames` or `clsx` used separately - Forms pattern: ```tsx const form = useForm({ resolver: zodResolver(schema) }); @@ -68,6 +71,8 @@ ## Accessibility - All interactive elements must have accessible labels (aria-label, aria-labelledby, or visible text) -- shadcn/ui + Radix provide WCAG 2.1 AA keyboard and screen reader support by default — do not override with custom handlers that break accessibility +- Accessible UI-primitive behavior (keyboard, focus, screen reader) is + governed by the `ux-radix` concern — do not override Radix/shadcn handlers + in ways that break the WAI-ARIA contract - Color contrast must meet WCAG AA (4.5:1 for normal text, 3:1 for large text) - Forms must associate labels with inputs and display validation errors accessibly diff --git a/workflows/concerns/resilience/practices.md b/workflows/concerns/resilience/practices.md index 33513fae..1e4a85a2 100644 --- a/workflows/concerns/resilience/practices.md +++ b/workflows/concerns/resilience/practices.md @@ -108,24 +108,11 @@ metrics/traces these guards emit into). They reference those concerns at the sea ## Boundary with neighbors -- **vs `enterprise-integration-patterns`**: EIP owns **async-channel** - resilience — a Dead Letter Channel for poison messages, an Idempotent Receiver - because an at-least-once *channel* will redeliver, Guaranteed Delivery against a - broker crash. This concern owns **synchronous-call** resilience — the timeout, - breaker, bulkhead, and fallback on a blocking in-the-call-path dependency. - **Idempotency is shared but split:** EIP's is the receiver's defense against - channel redelivery; this concern's is the **precondition that makes a - synchronous auto-retry safe.** When the boundary is a queue/broker, use EIP; - when it is a blocking synchronous call, use resilience. Do not duplicate. -- **vs `verification`**: this concern states the behavior that must be true (the - breaker opens, the timeout fires, the fallback serves); `verification` is the - gate that refuses "done" until those guard branches were **exercised against the - running system**. Hand the open/half-open and timeout paths to the verification - evidence gate; do not restate the gate here. -- **vs `o11y-otel`**: this concern requires guard activity to be **observable**; - `o11y-otel` owns the metric/trace/log plumbing that carries it. Emit - breaker-state and timeout/shed events into that pipeline; do not re-specify the - pipeline. +See `concern.md` for the canonical Boundary (vs `enterprise-integration-patterns`, +`verification`, `o11y-otel`). These practices stay on synchronous-call +stability; defer to the neighbor named there for async-channel resilience +(EIP), the evidence gate that proves the guards ran (`verification`), and the +telemetry pipeline that carries breaker/timeout/shed signals (`o11y-otel`). ## Quality Gates diff --git a/workflows/concerns/rust-cargo/concern.md b/workflows/concerns/rust-cargo/concern.md index 8227165a..3b952e85 100644 --- a/workflows/concerns/rust-cargo/concern.md +++ b/workflows/concerns/rust-cargo/concern.md @@ -62,5 +62,3 @@ additional deny-level lints. Selecting this concern requires these artifacts to change (a selected concern absent from them is drift): - ADR: Rust + Cargo workspace (clippy, fmt, cargo-deny/machete, pinned toolchain) as the language-runtime - TD: workspace lints, error-handling (thiserror/anyhow), unsafe policy, profile conventions - -## ADR References diff --git a/workflows/concerns/sample-data/practices.md b/workflows/concerns/sample-data/practices.md index e1787c11..593ff1a0 100644 --- a/workflows/concerns/sample-data/practices.md +++ b/workflows/concerns/sample-data/practices.md @@ -5,7 +5,7 @@ with — the data that makes it feel real and drives every UI state. They do not govern test data: `testing` owns fixtures, factories, and assertion-specific data (see the boundary in `concern.md`). -## Choose the faker (follows the tech stack) +## Frame - Use the **semantic fake-data library** that is the named default for the active runtime concern: @@ -16,7 +16,7 @@ data (see the boundary in `concern.md`). - Add it as a project dependency at a **pinned version**. The library is the default generator, not the source of determinism. -## Make generation deterministic +## Design Determinism comes from how the generator is driven, not from which one: @@ -27,9 +27,6 @@ Determinism comes from how the generator is driven, not from which one: - Generate in a **deterministic order** — no reliance on map/set iteration order or wall-clock time. - Record/print the seed so any dataset can be reproduced exactly. - -## Generate varied shapes and edge cases - - Produce **varied** rows — vary string lengths, optional-field presence, value ranges, and timestamps. Never N copies of one row. - Cover the **schema/domain-relevant edge cases** so every UI state renders: @@ -42,7 +39,7 @@ Determinism comes from how the generator is driven, not from which one: - Keep edge cases **bounded** — realistic extremes the schema and domain allow, not malformed data the product need not accept. -## Treat the seed script as a governed deliverable +## Build - The seed script is **idempotent and re-runnable**: running it again converges to the same governed dataset rather than appending duplicates. @@ -56,14 +53,7 @@ Determinism comes from how the generator is driven, not from which one: inside the same governed script for boundary cases you must guarantee. Avoid thin hardcoded-only data as the *whole* dataset. -## Boundary with testing - -- `sample-data` populates the **running product**; `testing` constructs - throwaway data for a single assertion. Do not seed the app from test fixtures, - and do not assert product behavior against the demo seed instead of - purpose-built test data. - -## Quality Gates +## Test - A semantic faker library is a pinned dependency for the data-backed product. - A governed, idempotent seed script exists and populates the running product. @@ -73,3 +63,12 @@ Determinism comes from how the generator is driven, not from which one: ordered generation. - Sample data is clearly synthetic (no PII) and the seed never targets production. + +## Cross-cutting + +### Boundary with testing + +- `sample-data` populates the **running product**; `testing` constructs + throwaway data for a single assertion. Do not seed the app from test fixtures, + and do not assert product behavior against the demo seed instead of + purpose-built test data. diff --git a/workflows/concerns/scala-sbt/concern.md b/workflows/concerns/scala-sbt/concern.md index 215ef478..99ae5531 100644 --- a/workflows/concerns/scala-sbt/concern.md +++ b/workflows/concerns/scala-sbt/concern.md @@ -40,5 +40,3 @@ maintain `scala-sbt` only for the remaining Scala surface. Selecting this concern requires these artifacts to change (a selected concern absent from them is drift): - ADR: Scala + sbt (scalafmt, scalafix, ScalaTest, ZIO where applicable) as the language-runtime - TD: sbt-dynver versioning, format/lint gates, build-cache and concurrency conventions - -## ADR References diff --git a/workflows/concerns/security-owasp/concern.md b/workflows/concerns/security-owasp/concern.md index 8e6acf2f..4315928b 100644 --- a/workflows/concerns/security-owasp/concern.md +++ b/workflows/concerns/security-owasp/concern.md @@ -6,6 +6,22 @@ security ## Areas all +## Boundary + +This concern owns the **hardening posture** for the product (OWASP Top 10: +injection, CSRF, secret handling, input validation, TLS, dependency auditing, +parameterized queries, error-detail leakage) and the **audit-logging policy** +for security-relevant events — what to log on authz denial, login failure, +privilege escalation. It is composable, applies across every area, and does +not fill a slot. + +For the family ownership table (auth / authorization-model / multi-tenancy / +security-owasp, plus the admin-console and unity-catalog neighbors) see +[README-auth-family.md](../README-auth-family.md). Broken Access Control is the +OWASP umbrella; `authorization-model` is the per-handler model that prevents +it, and `multi-tenancy` is the tenant-predicate refinement — neither is +restated here. + ## Components - **Standard**: OWASP Top 10 (current edition) diff --git a/workflows/concerns/twelve-factor/practices.md b/workflows/concerns/twelve-factor/practices.md index db54f234..da11e193 100644 --- a/workflows/concerns/twelve-factor/practices.md +++ b/workflows/concerns/twelve-factor/practices.md @@ -88,15 +88,11 @@ at the codebase, the release artifact, or the runbook and confirm or reject it. ## Stay in your lane (boundary with sibling concerns) -- Do **not** decide how many deployables the system ships as or where their - seams fall here — that is `deployment-topology`. This concern is the contract - **each** deployable honors. -- Do **not** specify log schema, span naming, trace context, or metric names - here — that is `o11y-otel`. This concern only says the event stream goes to - **stdout** and the process does not manage log files. -- Do **not** review cluster / Helm / image-build / secret-injection mechanics - here — that is `k8s-kind` / the `deploy-target` filler. This concern produces - the process the runtime consumes; it does not specify the runtime. +See `concern.md` for the canonical Boundary (vs `deployment-topology`, +`o11y-otel`, `k8s-kind` / the `deploy-target` filler). These practices are +the per-process operational contract — defer to the neighbor named there for +the deployable count and seams, the log/metric/trace schema, and the cluster +/ Helm / image-build mechanics. ## Artifact impact (what selecting this changes) diff --git a/workflows/concerns/unity-catalog/concern.md b/workflows/concerns/unity-catalog/concern.md index 4069d4b9..4b2bb70d 100644 --- a/workflows/concerns/unity-catalog/concern.md +++ b/workflows/concerns/unity-catalog/concern.md @@ -17,30 +17,17 @@ data-modeling or access-control concern (see `## Boundary`). This concern owns **how data and AI assets are governed on Databricks** — the catalog namespace, the grant model, lineage, and governed external storage. It -is **Databricks' concrete realization** of data governance and must not be -restated as, or used to duplicate, the generic neighbors: - -- **Generic data modeling / `domain-driven-design`** owns *what the data - means* — entities, aggregates, invariants, the logical schema. Unity Catalog - owns *where those assets are registered and who may touch them* — the physical - `catalog.schema.object` namespace and the GRANT model over it. Model the - domain in the domain concern; register and govern it here. Do not re-derive - the logical model in this concern. -- **`security-owasp` / generic authz** owns *application-layer* authentication - and authorization (sessions, RBAC in app code, the OWASP risks). Unity Catalog - owns *data-layer* governance enforced by the platform at query time - (privileges, row filters, column masks, ownership). A Databricks App still - authenticates its users at the app layer; Unity Catalog is the additional - governance the *data* access flows through. Do not collapse the two — app - RBAC is not a substitute for catalog grants, and catalog grants are not a - substitute for app authz. -- **`databricks-declarative-pipelines`** *produces* governed datasets (its - streaming tables and materialized views land in a catalog/schema); Unity - Catalog *governs* them. The pipeline declares the dataset; this concern owns - the grants, ownership, and lineage on the result. -- **`databricks-apps`** *consumes* governed data; Unity Catalog *governs* the - consumption. This concern owns the rule that an app reads through Unity - Catalog grants, not around them. +is **Databricks' concrete realization** of data governance. + +For the auth family (where app-layer `authorization-model` and catalog grants +**compose** — neither substitutes for the other), see +[README-auth-family.md](../README-auth-family.md). For the logical domain +model, defer to `domain-driven-design`: model entities/aggregates there; +register and govern the physical `catalog.schema.object` namespace here. +`databricks-declarative-pipelines` *produces* governed datasets; this concern +owns the grants, ownership, and lineage on the result. `databricks-apps` +*consumes* governed data; this concern owns the rule that an app reads +through Unity Catalog grants, not around them. ## Components diff --git a/workflows/concerns/unity-catalog/practices.md b/workflows/concerns/unity-catalog/practices.md index a08d3af0..4afa7bce 100644 --- a/workflows/concerns/unity-catalog/practices.md +++ b/workflows/concerns/unity-catalog/practices.md @@ -2,9 +2,10 @@ These practices govern **how data and AI assets are registered, granted, and lineage-tracked on Databricks**. They are the Databricks realization of data -governance; they do not restate the logical data model (`domain-driven-design`) -or application-layer authz (`security-owasp`) — see the boundary in -`concern.md`. +governance. For the boundary (composition with `authorization-model` / +`security-owasp`, `domain-driven-design`, `databricks-apps`, +`databricks-declarative-pipelines`) see `concern.md` and the auth family +ownership table at [README-auth-family.md](../README-auth-family.md). ## Requirements (Frame activity) @@ -55,19 +56,6 @@ or application-layer authz (`security-owasp`) — see the boundary in - Verify **lineage** is captured for the product's key tables (upstream → downstream visible in Unity Catalog lineage). -## Boundary with neighbors - -- **vs domain modeling (`domain-driven-design`)**: model entities/aggregates in - the domain concern; register and govern the physical namespace here. Do not - re-derive the logical model in catalog terms. -- **vs `security-owasp` / app authz**: app-layer auth and catalog grants - **compose**; neither substitutes for the other. App RBAC does not replace - `SELECT`/`MODIFY` grants; catalog grants do not replace app authentication. -- **vs `databricks-declarative-pipelines`**: the pipeline declares the dataset; - this concern owns the grants/ownership/lineage on the result. -- **vs `databricks-apps`**: the app consumes data; this concern owns the rule - that it consumes **through** Unity Catalog. - ## Quality Gates - All tables, views, volumes, and models the product uses are **registered in diff --git a/workflows/concerns/ux-radix/concern.md b/workflows/concerns/ux-radix/concern.md index 4980ca59..0143d6db 100644 --- a/workflows/concerns/ux-radix/concern.md +++ b/workflows/concerns/ux-radix/concern.md @@ -8,7 +8,12 @@ ui, frontend ## Components -- **Primitives**: Radix UI (headless, accessible by default) +- **Primitives**: Radix UI (headless, accessible by default) — this concern is + the canonical owner of the Radix prescription; other concerns reference here +- **Component library**: shadcn/ui (copied components, not an npm dependency) + built on top of Radix — this concern is the canonical owner of the shadcn + prescription; framework concerns (e.g. `react-nextjs`) reference here rather + than re-prescribing - **Patterns**: WAI-ARIA design patterns for all interactive widgets - **Scope**: Searching, editing, navigation, selection, and disclosure @@ -109,12 +114,17 @@ Any project with interactive user interfaces that involve searching, editing, navigating, or selecting data. Composes with `a11y-wcag-aa` for compliance requirements and `react-nextjs` for React-specific implementation patterns. Framework-agnostic in principle — the patterns are WAI-ARIA standards; Radix -is the reference implementation for React projects. +is the reference implementation for React projects, and shadcn/ui is the +prescribed component library that wraps Radix for React projects. ## Artifact Impact Selecting this concern requires these artifacts to change (a selected concern absent from them is drift): -- DESIGN_SYSTEM: WAI-ARIA interaction patterns for search/edit/nav/select/disclosure; Radix primitives; states +- ADR: shadcn/ui + Radix as the UI-primitives prescription (component library copied, not npm dep) +- DESIGN_SYSTEM: WAI-ARIA interaction patterns for search/edit/nav/select/disclosure; Radix primitives; shadcn component conventions; states - TEST_PLAN: keyboard navigation, focus-return, active-state, and live-filter behavior checks ## ADR References + +- ADR-011: ux-radix owns the Radix and shadcn component-library prescription; + framework concerns reference rather than re-prescribe diff --git a/workflows/concerns/ux-radix/practices.md b/workflows/concerns/ux-radix/practices.md index 6f9fcdeb..c9496852 100644 --- a/workflows/concerns/ux-radix/practices.md +++ b/workflows/concerns/ux-radix/practices.md @@ -76,6 +76,22 @@ carry every state: ## Implementation +### Component library (canonical owner) + +This concern is the canonical owner of the Radix and shadcn/ui prescription. +Framework concerns (e.g. `react-nextjs`) MUST reference this concern for UI +primitives instead of re-prescribing shadcn or Radix themselves. + +- Use **shadcn/ui** as the component library, built on Radix primitives. +- shadcn components are **copied into the project** (e.g. via + `npx shadcn@latest add `) and customized in `components/ui/` — + shadcn is NOT installed as an npm dependency. +- Customize copied shadcn components to match the project's design tokens; + keep the Radix-based accessibility behavior intact (do not override keyboard + or focus handlers with custom logic that breaks the WAI-ARIA contract). +- For headless behavior not covered by shadcn, use the underlying Radix + primitive directly. + ### Radix component mapping | Pattern | Radix Primitive | Notes |