You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
docs: update v3.6.1 to match Option A — three sibling dirs, per-kernel repos, CKProject CRD
Rewrote versioning.md for Option A (runc constraint discovery, no /ck-tool/ root,
per-kernel bare repos, three sibling dirs under kubelet namespace dir).
Added operator.md sections: per-version deployments, three PVs per kernel,
CKProject CRD with kubectl get ckp, kopf+NATS dual control plane, .status patching,
versioned reconciliation lifecycle (12 steps).
Updated changelog.md v3.6.1 entry with all findings.
Updated introduction.md release train with PROVEN status.
Version materialisation moves from `serving.json` on disk to the CK.Project custom resource. The CK volume becomes purely immutable -- no write-through exceptions.
48
+
Version materialisation moves from `serving.json` on disk to the CK.Project custom resource. The CK volume becomes purely immutable -- no write-through exceptions. Storage uses per-kernel bare repositories and three sibling directories per kernel version (Option A).
49
49
50
50
### What Changes
51
51
52
52
-**serving.json retired** -- no longer exists on disk. The three problems it created (write-through hack, inert file, decorative git refs) are dissolved.
53
-
-**CK.Project `spec.versions`** -- named versions with git commit ref and URL route prefix. Version state is in etcd, not the filesystem.
54
-
-**`spec.repo`** -- upstream git URL. The operator clones to scratch space and runs `git archive` locally (FUSE mounts cannot host bare repos).
55
-
-**`deploy.materialise` step** -- new reconciliation step between `deploy.namespace` and `deploy.storage`. Streams `git archive` output to filer at `/ck/v/{tag}/{kernel}/`.
56
-
-**`.git-ref` stamp** -- each materialised directory contains the exact commit hash. Implements `prov:wasGeneratedBy` from Git2PROV.
57
-
-**Per-version PVs** -- `ck-{project}-{kernel}-{tag}`, each pointing to `/ck/v/{tag}/{kernel}`. When a tag's ref changes, content updates without PV recreation.
53
+
-**Option A: Three sibling dirs** -- in-container mount layout uses `/ck/{kernel}/ck/`, `/ck/{kernel}/tool/`, `/ck/{kernel}/data/` as three sibling PVs under a kubelet-created namespace directory. No nested volume mounts. Driven by the runc constraint: `mkdirat` fails with EROFS in ReadOnly parent overlays before CSI volume content is visible.
54
+
-**Per-kernel bare repos** -- each kernel has its own isolated bare git repository at `/ck/{kernel}/` on the SeaweedFS filer. No monorepo. No `spec.repo`. CK and TOOL loops extract from the same bare repo to sibling `ck/` and `tool/` directories.
55
+
-**CKProject CRD** -- `ck.tech.games/v1` kind `CKProject` with `spec.versions` containing per-kernel `ck_ref` and `tool_ref`. `kubectl get ckp -A` shows Phase, Versions, and Checks.
56
+
-**kopf + NATS dual control plane** -- both kopf CRD watch and NATS message listener trigger the same `reconcile()` function. `kubectl apply` and `nats pub` produce identical results.
57
+
-**CKProject .status patching** -- operator patches `.status` after each reconcile with phase, per-version materialisation state, and aggregate proof.
58
+
-**`deploy.materialise` step** -- new reconciliation step between `deploy.namespace` and `deploy.storage`. Extracts CK loop to `/ck/{kernel}/{version}/ck/` and TOOL loop to `/ck/{kernel}/{version}/tool/` via `git archive` from per-kernel bare repos.
59
+
-**Three PVs per kernel per version** -- `ck-{project}-{kernel}-{version}-ck`, `ck-{project}-{kernel}-{version}-tool`, `ck-{project}-{kernel}-{version}-data`. CK and TOOL ReadOnlyMany, DATA ReadWriteMany.
60
+
-**Per-version deployments** -- each declared version gets its own Deployment and service. Multiple versions run simultaneously at different route prefixes.
58
61
-**Per-version HTTPRoutes** -- longest-prefix-first routing. Each version gets its own deployment and route rule.
59
-
-**Garbage collection** -- filer paths under `/ck/v/` not referenced by any CK.Project version are deleted after reconciliation.
60
-
-**Shared kernel optimisation** -- identical tree hashes across versions share one filer copy.
62
+
-**`.git-ref` stamp** -- each materialised loop directory (`ck/` and `tool/`) contains the exact commit hash. Implements `prov:wasGeneratedBy` from Git2PROV.
63
+
-**Quick setup mode** -- no git required for bootstrapping. Omit `ck_ref`/`tool_ref` and upload files directly to the filer. Transition to git-managed later without changing PVs or mounts.
64
+
-**Garbage collection** -- version directories under `/ck/{kernel}/` not referenced by any CK.Project are deleted. Git internals (`HEAD`, `objects/`, `refs/`) are never GC'd.
65
+
-**Shared kernel optimisation** -- identical refs across versions share one filer copy, PVs point to existing path.
61
66
-**Backward compatible** -- no `spec.versions` means flat layout, existing projects unchanged.
62
-
63
-
### Git Model (D9)
64
-
65
-
- One repo per project. CK + TOOL in same repo, DATA never in git.
66
-
- Tag prefix convention: `ck/{kernel}/vX.Y.Z`, `tool/{kernel}/vX.Y.Z` for independent loop versioning.
67
-
- Per-loop overrides: `ck_ref` and `tool_ref` in version declarations for split CK/TOOL tags.
68
-
- Ontological grounding: DOAP for repo metadata, Git2PROV (PROV-O) for commit provenance.
67
+
-**Proven on live cluster** -- `hello-v1-0-0-proc` Running with three sibling PVs on AKS + SeaweedFS 3.93.
@@ -81,7 +81,7 @@ For the full step-by-step breakdown, see [Reconciliation Lifecycle](./reconcilia
81
81
## Version Materialisation (v3.6.1)
82
82
83
83
::: info v3.6.1 -- serving-multiversion-unpack
84
-
This section documents the version materialisation model introduced in v3.6.1, implemented in CK.Operator v1.3.0. See [Versioning](./versioning) for the full git model.
84
+
This section documents the version materialisation model introduced in v3.6.1, implemented in CK.Operator v1.3.0. See [Versioning](./versioning) for the full storage layout and runc constraint analysis.
85
85
:::
86
86
87
87
### serving.json Is Retired
@@ -96,110 +96,202 @@ v3.6.1 dissolves all three problems by moving version state from the filesystem
96
96
97
97
### Version State in CK.Project CR
98
98
99
-
Version declarations live in the CK.Project custom resource:
99
+
Version declarations live in the CK.Project custom resource. Each version specifies per-kernel git refs for both CK and TOOL loops:
The operator reads `spec.versions`, materialises each version from per-kernel bare repositories on the SeaweedFS filer, and creates per-version deployments, PVs, and HTTPRoute rules. A version promotion is a CK.Project resource update -- `kubectl patch`, NATS command, or operator API. Standard Kubernetes-native workflow with etcd history.
132
+
133
+
### Per-Version Deployments
134
+
135
+
Each declared version gets its own Deployment. Pods in one version's deployment mount that version's PVs only. Multiple versions of the same project run simultaneously, each serving its designated route:
Inside the pod, these mount as three sibling directories under the kernel name:
155
+
156
+
```
157
+
/ck/{kernel}/ck/ ← CK PV (ReadOnly)
158
+
/ck/{kernel}/tool/ ← TOOL PV (ReadOnly)
159
+
/ck/{kernel}/data/ ← DATA PV (ReadWrite)
160
+
```
161
+
162
+
The kernel name directory is not a volume -- it is a plain directory created by the kubelet. See [Versioning -- Three Sibling Dirs](./versioning#in-container-mount-layout-three-sibling-dirs-option-a) for the runc constraint that drives this design.
163
+
164
+
### CKProject CRD
165
+
166
+
v3.6.1 introduces the `CKProject` Custom Resource Definition at `ck.tech.games/v1`:
167
+
168
+
```bash
169
+
kubectl get ckp -A
170
+
```
171
+
172
+
```
173
+
NAMESPACE NAME PHASE VERSIONS CHECKS AGE
174
+
ck-hello hello Running 1 15 1d
175
+
ck-delvinator delvinator Running 1 15 2d
176
+
```
177
+
178
+
The CKProject CRD provides:
179
+
180
+
- **`spec.versions`** -- array of named versions with per-kernel `ck_ref`/`tool_ref`
181
+
- **`spec.hostname`** -- project hostname for HTTPRoute generation
182
+
- **`.status.phase`** -- Running, Degraded, or Failed (patched by operator after reconcile)
183
+
- **`.status.versions`** -- per-version status with materialisation state
184
+
- **`.status.proof`** -- aggregate proof from reconciliation checks
185
+
186
+
#### CKProject .status Patching
187
+
188
+
After each reconciliation, the operator patches the CKProject `.status` with the reconciliation result:
189
+
190
+
```yaml
191
+
status:
192
+
phase: Running
193
+
versions:
194
+
- name: v1.0.0
195
+
materialised: true
196
+
deploymentReady: true
197
+
routeAccepted: true
198
+
proof:
199
+
totalChecks: 15
200
+
totalPassed: 15
201
+
lastReconciled: "2026-04-06T14:00:00Z"
114
202
```
115
203
116
-
The operator reads `spec.versions`, materialises each version from the git repository, and creates per-version PVs and HTTPRoute rules. A version promotion is a CK.Project resource update -- `kubectl patch`, NATS command, or operator API. Standard Kubernetes-native workflow with etcd history.
204
+
This means `kubectl get ckp` always reflects the actual cluster state. If materialisation fails or a deployment is unhealthy, the status shows it.
205
+
206
+
### Dual Control Plane: kopf + NATS
207
+
208
+
CK.Operator uses both kopf (Kubernetes watch) and NATS messaging as entry points. Both trigger the same `reconcile()` function:
209
+
210
+
```
211
+
kopf watch CKProject CR change ──→ reconcile()
212
+
NATS input.CK.Operator message ──→ reconcile()
213
+
```
214
+
215
+
This provides two equivalent paths to the same outcome:
- Writes commit hashes to `.git-ref` in each loop directory
255
+
4. Ensures three PVs per kernel per version exist
256
+
5. Ensures PVCs are bound
148
257
149
-
### Per-Version PVs and HTTPRoutes
150
-
151
-
Each declared version gets its own PersistentVolume and routing rule:
152
-
153
-
```yaml
154
-
# Per-version CK volume
155
-
apiVersion: v1
156
-
kind: PersistentVolume
157
-
metadata:
158
-
name: ck-delvinator-core-live
159
-
spec:
160
-
accessModes: [ReadOnlyMany]
161
-
capacity:
162
-
storage: 50Gi
163
-
csi:
164
-
driver: seaweedfs-csi-driver
165
-
volumeHandle: ck-delvinator-core-live
166
-
volumeAttributes:
167
-
path: "/ck/v/live/Delvinator.Core"
168
-
```
258
+
### Per-Version HTTPRoutes
169
259
170
-
HTTPRoute rules are ordered longest-prefix first, so `/staging` matches before `/`:
260
+
HTTPRoute rules are ordered longest-prefix first:
171
261
172
262
```yaml
173
263
spec:
174
-
hostnames: [delvinator.tech.games]
264
+
hostnames: [hello.tech.games]
175
265
rules:
176
266
- matches:
177
-
- path: { type: PathPrefix, value: /staging }
267
+
- path: { type: PathPrefix, value: /next }
178
268
backendRefs:
179
-
- name: delvinator-staging
269
+
- name: hello-v1.3.19
270
+
port: 80
180
271
- matches:
181
272
- path: { type: PathPrefix, value: / }
182
273
backendRefs:
183
-
- name: delvinator-live
274
+
- name: hello-v1.3.2
275
+
port: 80
184
276
```
185
277
186
-
When a tag's ref changes, the PV is not recreated -- the filer path is stable (keyed by tag name). Content changes when the operator re-materialises. Pods pick up new content on rolling restart.
278
+
When a version's ref changes, the PV is not recreated -- the filer path is stable (keyed by version name). Content changes when the operator re-materialises. Pods pick up new content on rolling restart.
After reconciliation, the operator scans `/ck/v/` on the filer and deletes directories not referenced by any active version in any CK.Project resource. GC logs every deletion and never removes directories referenced by other projects.
290
+
After reconciliation, the operator scans version directories under each kernel on the filer and deletes those not referenced by any active version in any CK.Project resource. GC deletes both `ck/` and `tool/` subdirectories together. GC MUST NOT delete git internals (`HEAD`, `objects/`, `refs/`). GC logs every deletion and never removes directories referenced by other projects.
199
291
200
292
### Shared Kernel Optimisation
201
293
202
-
When two versions reference the same git tree hash for a kernel (e.g., the CK loop has not changed between live and staging), the operator reuses one filer copy. The PV for the second versionpoints to the first version's filer path -- no duplicate storage.
294
+
When two versions reference the same commit for a kernel's loop, the operator does not extract twice. It checks `.git-ref` in the existing version directory. If the hash matches, the new version's PV points to the existing path -- no duplicate storage.
0 commit comments