|
| 1 | +# GitOps: two source-of-truth modes (K2 design) |
| 2 | + |
| 3 | +**Status:** design (invariants agreed; code reconciliation is the follow-up). |
| 4 | +**Owner:** roadmap K2. Depends on K3 (provenance `source` column — shipped). |
| 5 | + |
| 6 | +## Problem |
| 7 | + |
| 8 | +openctl's git integration grew in two directions that now overlap: |
| 9 | + |
| 10 | +- **git-as-sink** (original): SQLite is the source of truth. After every apply |
| 11 | + the controller *materializes* the desired spec to a disk mirror |
| 12 | + (`manifests.dir`), auto-commits it (`manifests.git`), and optionally pushes. |
| 13 | + On startup it *re-materializes* any missing mirror files so the audit trail |
| 14 | + is complete. |
| 15 | +- **git-as-source** (B1–B3): the repo is the source of truth. The controller |
| 16 | + *pulls* a remote (`manifests.gitops.pull`), applies changed files, *prunes* |
| 17 | + resources whose file left the repo (`pull.prune`), and converges immediately |
| 18 | + on a push webhook. |
| 19 | + |
| 20 | +Both sets of machinery can be enabled at once, and they interact badly: |
| 21 | + |
| 22 | +1. **Startup re-materialize fights prune.** Prune deletes resources whose file |
| 23 | + was removed from the repo; startup re-materialize re-creates mirror files |
| 24 | + from SQLite — so a pruned resource's file can reappear and the resource comes |
| 25 | + back. |
| 26 | +2. **`--ff-only` pull fights auto-commit.** git-as-source pulls with |
| 27 | + `--ff-only`; git-as-sink auto-commits desired specs on every apply. A local |
| 28 | + auto-commit makes the next `--ff-only` pull fail (diverged history). |
| 29 | +3. **Ambiguous truth.** With both on, "what should exist" has two authorities |
| 30 | + (SQLite and the repo) with no rule for which wins. |
| 31 | + |
| 32 | +The machinery — disk mirror, auto-commit, push, pull, watcher, prune, webhook, |
| 33 | +plus the always-on drift reconciler — is individually fine; the problem is that |
| 34 | +nothing says *which combination is coherent*. |
| 35 | + |
| 36 | +## Goal |
| 37 | + |
| 38 | +Collapse the combinations into **two explicit, non-overlapping modes**, selected |
| 39 | +by **one config switch**, where only the machinery for the chosen mode runs. |
| 40 | +Each mode has a single, unambiguous source of truth and no self-conflicting |
| 41 | +components. |
| 42 | + |
| 43 | +Non-goals: changing the reconcile/apply pipeline itself; changing the drift |
| 44 | +reconciler (it runs in both modes); a UI for switching modes. |
| 45 | + |
| 46 | +## The two modes |
| 47 | + |
| 48 | +| Aspect | **mirror** (SQLite is truth) | **gitops** (the repo is truth) | |
| 49 | +|---|---|---| |
| 50 | +| Source of truth | SQLite `applied_manifests` | the git repo (working tree in `manifests.dir`) | |
| 51 | +| git role | read-only **audit mirror** of what was applied | the **input** — authored externally, pulled in | |
| 52 | +| Materialize desired spec → disk on apply | **yes** | **no** (the file already exists in the repo) | |
| 53 | +| Auto-commit on apply/delete | **yes** | **no** (would fight `--ff-only` pull) | |
| 54 | +| Push to remote | **yes** (`onCommit`/`periodic`/`manual`) | **no** (the repo is authored upstream) | |
| 55 | +| Startup: re-materialize missing files | **yes** (audit completeness) | **no** (prune must be able to remove them) | |
| 56 | +| Pull remote → apply | **no** | **yes** (interval + webhook) | |
| 57 | +| Prune (repo = desired SET) | **no** | **yes** (opt-in; heavily guarded) | |
| 58 | +| Push webhook | **no** | **yes** (opt-in) | |
| 59 | +| fsnotify local file → apply | **no** | **yes** (edit a file, it applies) | |
| 60 | +| Drift reconciler (periodic re-apply of stored desired) | **yes** | **yes** | |
| 61 | + |
| 62 | +`mirror` is today's default behavior (one-way sink + optional push). `gitops` |
| 63 | +is the DriftlessAF loop. |
| 64 | + |
| 65 | +## Invariants |
| 66 | + |
| 67 | +1. **Exactly one mode is active** for a given controller. No "both." |
| 68 | +2. **mirror never reads from git.** No pull, no prune, no webhook, no |
| 69 | + file→apply. git is written, never read as intent. |
| 70 | +3. **gitops never writes desired specs to git.** No auto-commit of applied |
| 71 | + manifests, no push of desired state. The controller may still write |
| 72 | + *operational* artifacts elsewhere (op history in SQLite), but the repo's |
| 73 | + desired specs are authored by humans/CI, not the controller. |
| 74 | +4. **gitops does not revive files on startup.** A file absent from the working |
| 75 | + tree is a signal ("this resource is no longer desired"), not a gap to fill. |
| 76 | + Reviving it would resurrect a pruned resource (conflict #1) — forbidden. |
| 77 | +5. **The drift reconciler is mode-agnostic.** It re-applies the *stored* desired |
| 78 | + state (`applied_manifests`) on drift, in both modes. In gitops mode that |
| 79 | + stored state is simply whatever the last pull/apply wrote — so the reconciler |
| 80 | + converges toward the repo, transitively, without a second code path. |
| 81 | +6. **gitops prune trusts provenance.** Prune only deletes resources whose last |
| 82 | + apply was `source=gitops` (or unknown), never `cli`/`ui`. This is now a |
| 83 | + durable column read (K3), not an ops-table reconstruction — so provenance |
| 84 | + survives op GC and prune stays safe. |
| 85 | + |
| 86 | +## Config |
| 87 | + |
| 88 | +Add one switch; keep the existing sub-blocks but scope them to a mode. |
| 89 | + |
| 90 | +```yaml |
| 91 | +manifests: |
| 92 | + dir: ~/.openctl/manifests |
| 93 | + mode: mirror # mirror (default) | gitops |
| 94 | + |
| 95 | + # mirror-mode settings (ignored in gitops mode): |
| 96 | + git: |
| 97 | + enabled: true # auto-commit the audit mirror |
| 98 | + remote: git@github.com:me/infra-audit.git |
| 99 | + pushMode: onCommit |
| 100 | + |
| 101 | + # gitops-mode settings (ignored in mirror mode): |
| 102 | + gitops: |
| 103 | + remote: git@github.com:me/infra.git # the source repo |
| 104 | + pull: |
| 105 | + interval: 1m |
| 106 | + prune: true |
| 107 | + webhook: |
| 108 | + enabled: true |
| 109 | + secret: { $secret: { provider: env, key: WEBHOOK_SECRET } } |
| 110 | +``` |
| 111 | +
|
| 112 | +- `mode` defaults to **`mirror`** — the current default behavior, so existing |
| 113 | + configs are unchanged. |
| 114 | +- **Contradictory combinations are rejected at config load** with a clear |
| 115 | + error, rather than silently half-working. Examples: |
| 116 | + - `mode: gitops` with `git.pushMode`/`git.enabled` set → error ("gitops mode |
| 117 | + does not auto-commit or push desired specs; remove `manifests.git`"). |
| 118 | + - `mode: mirror` with `gitops.pull` set → error ("pull/prune require |
| 119 | + `mode: gitops`"). |
| 120 | + - `mode: gitops` with no `gitops.remote` → error. |
| 121 | + |
| 122 | +### Back-compat / migration |
| 123 | + |
| 124 | +- Absent `mode` → `mirror`. A config that only uses `manifests.git` keeps |
| 125 | + working verbatim. |
| 126 | +- A config that currently sets `manifests.gitops.pull.enabled: true` is the one |
| 127 | + breaking case: on load, if `pull` is configured without `mode: gitops`, emit |
| 128 | + a one-line deprecation error telling the operator to set `mode: gitops` (and |
| 129 | + drop any `manifests.git` push settings). This is deliberate — that config was |
| 130 | + the ambiguous one this change exists to disambiguate. |
| 131 | + |
| 132 | +## Code reconciliation plan (the follow-up) |
| 133 | + |
| 134 | +Small, mechanical once the invariants above are fixed. Per component: |
| 135 | + |
| 136 | +1. **Config** (`internal/config`): add `Manifests.Mode` (default `mirror`); add |
| 137 | + `validateManifestsMode()` rejecting the contradictory combos; move |
| 138 | + `gitops.remote` alongside (today the remote lives on `manifests.git`). |
| 139 | +2. **Wiring** (`cmd/openctl-controller/main.go`): branch on `mode`. Wire the |
| 140 | + disk-mirror auto-commit + push + startup-revive **only** in mirror mode; wire |
| 141 | + the watcher + pull loop + pruner + webhook **only** in gitops mode. The |
| 142 | + drift reconciler is wired unconditionally. |
| 143 | +3. **DiskMirror** (`internal/controller/manifests/disk.go`): gate the startup |
| 144 | + re-materialize behind mirror mode (a constructor flag or a |
| 145 | + `mode`-aware caller). In gitops mode the mirror dir is the *working tree*, |
| 146 | + not a controller-owned output — so nothing re-materializes. |
| 147 | +4. **git hook** (`internal/controller/manifests/githook.go`): auto-commit only |
| 148 | + in mirror mode. |
| 149 | +5. **Watcher / Repo / Pruner / Webhook**: unchanged internally; simply not |
| 150 | + constructed in mirror mode. (They already exist and are individually tested.) |
| 151 | +6. **No change** to the dispatcher, drift reconciler, or `applied_manifests` |
| 152 | + store — the store stays the desired-state cache in both modes; K3's `source` |
| 153 | + column already distinguishes provenance. |
| 154 | + |
| 155 | +## Test plan |
| 156 | + |
| 157 | +- **Config validation**: each contradictory combination errors with the |
| 158 | + documented message; each valid single-mode config loads; absent `mode` |
| 159 | + resolves to `mirror`. |
| 160 | +- **Mode selection**: a fake wiring harness asserts that in mirror mode the |
| 161 | + pull/prune/webhook components are nil and the mirror/commit ones are live, and |
| 162 | + vice-versa in gitops mode. |
| 163 | +- **Conflict #1 gone**: in gitops mode, deleting a file + pull → the resource is |
| 164 | + pruned and startup does **not** revive its file. |
| 165 | +- **Conflict #2 gone**: in gitops mode, an apply does **not** auto-commit, so a |
| 166 | + subsequent `--ff-only` pull succeeds. |
| 167 | +- Existing per-component tests (mirror, watcher, pruner, webhook) continue to |
| 168 | + pass unchanged. |
| 169 | + |
| 170 | +## Open question (ties to K5) |
| 171 | + |
| 172 | +Which mode is the *recommended* default is partly a product-scope question (K5: |
| 173 | +is openctl infra-IaC-with-bounded-workloads, or a homelab PaaS?). This doc |
| 174 | +fixes the *mechanism* regardless: `mirror` stays the safe default (no surprise |
| 175 | +deletes, no external repo required), and `gitops` is the opt-in DriftlessAF |
| 176 | +loop. The scope decision only changes which mode the docs lead with, not the |
| 177 | +invariants here. |
0 commit comments