Skip to content

feat(sync): multiple disjoint sync mappings, each with its own direction#156

Merged
acmore merged 4 commits into
mainfrom
feat/sync-multi-folder
Jul 5, 2026
Merged

feat(sync): multiple disjoint sync mappings, each with its own direction#156
acmore merged 4 commits into
mainfrom
feat/sync-multi-folder

Conversation

@acmore

@acmore acmore commented Jul 5, 2026

Copy link
Copy Markdown
Owner

Completes the multi-folder sync capability on top of #155's per-path direction schema: spec.sync.paths now accepts multiple mappings, each becoming its own syncthing folder with its own direction.

sync:
  paths:
    - .:/workspace                   # primary: code, bi (or up)
    - local: ./collected
      remote: /data/results
      direction: down                # pod-generated results, pod is authority
    - local: ../datasets
      remote: /data/datasets
      direction: up                  # dataset push, local is authority

Rules

  • Disjoint-only: mapping roots must be pairwise disjoint on both sides (equal or nested roots rejected at validation, lexical check). Same-root folders are impossible in syncthing (.stignore lives in the directory); nested roots would sync a subtree through two folders and reintroduce the overwrite hazards directions exist to prevent.
  • Primary mapping (first entry): keeps the legacy okdev-<session> folder ID so existing sessions upgrade in place; it is the folder fanned out to mesh receivers; it is the only remote cleared by up --reset-workspace / sync reset-remote (a note is printed when extra mappings exist). Additional mappings stay local↔hub only — fleet-wide result collection remains jobs logs/cp.
  • Stable IDs: extra mappings get okdev-<session>-<sha8(local:remote)>, index-independent so reordering doesn't reset folders.
  • Cleanup: mappings removed from the config get their folders pruned from both syncthing instances on the next start; only exact managed ID shapes are touched (other sessions' and user-defined folders never removed — unit-tested).
  • Modes: default = each folder syncs in its mapping's direction; explicit sync --mode forces all folders; the two-phase bootstrap applies per folder to bi mappings only. The sync config hash covers all pairs/directions and hashes single-mapping bi sessions identically to the legacy shape — upgrading okdev does not restart existing sessions.

Testing

  • Unit: overlap validation (equal/nested/disjoint/lexical edges), per-path mode resolution incl. override and two-phase scoping, folder ID stability, prune safety, multi-pair config hash (incl. legacy-identical single-bi), ready-card/summary arrows.
  • Kind e2e (new scenarios on the live smoke session): okdev validate rejects overlapping mappings; adding a second disjoint mapping (code up + results down) takes effect via config-hash restart; both folders sync in their own directions; status --details shows per-path arrows.
  • gofmt -l clean; go test -race ./... passes.

🤖 Generated with Claude Code

acmore and others added 4 commits July 5, 2026 19:32
Lift the single-mapping limit for the syncthing engine: spec.sync.paths
now accepts multiple mappings as long as their roots are pairwise
disjoint on both sides (equal or nested roots are rejected — two
syncthing folders cannot share a root since .stignore lives in the
directory, and nested roots would sync a subtree through two folders).

Each mapping becomes its own syncthing folder. The first entry is the
primary mapping: it keeps the legacy "okdev-<session>" folder ID so
existing sessions upgrade in place, it is the folder shared to mesh
receivers, and it is the only remote cleared by up --reset-workspace /
sync reset-remote (with a printed note when extra mappings exist).
Additional mappings get stable IDs derived from the pair (reordering
does not reset their folders) and stay local<->hub only. Mappings
removed from the config have their folders pruned from both syncthing
instances on the next start (managed-ID shapes only, other sessions'
and user folders untouched).

Mode semantics follow: by default every folder syncs in its mapping's
direction; an explicit `sync --mode` forces all folders; `okdev up`
passes only two-phase-or-empty, and the two-phase bootstrap applies per
folder to bi mappings (directional folders configure their final types
directly). The sync config hash covers all pairs and directions while
hashing single-mapping bi sessions identically to the legacy shape, so
upgrades do not restart existing sessions. Ready card, status --details
configured paths, and dry-run output render per-path direction arrows.

Kind e2e covers: overlap rejection via okdev validate, adding a second
disjoint mapping to a live session (code up + results down), both
folders syncing in their own directions, and per-path arrows in status.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…roots

The kind e2e exposed that additional sync mappings could never work: the
remote syncthing runs in the sidecar, which only mounted the workspace
volume, so any extra remote root lived on the dev container's overlay —
invisible to the sidecar, silently "syncing" into a directory the
workload never reads.

- PreparePodSpec now mirrors the target container's volume mounts into
  the sidecar (same paths, dedup against sidecar internals), so any
  volume-backed remote root is servable by the sidecar's syncthing.
- Sync startup probes each additional mapping's remote root: the
  sidecar writes a marker and the target container must see it;
  otherwise startup fails with guidance to mount a volume and re-run
  `okdev up --reconcile` instead of silently splitting the two
  containers' views.
- Docs call out the volume requirement, and the invalid nested-local
  examples ("./collected" under ".") are corrected to siblings — the
  disjoint rule applies to the local side too.
- Kind e2e adds a results-vol emptyDir mounted at /data and reconciles
  before asserting the second mapping syncs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…o-phase

`okdev up` waits for the bootstrap-complete handoff whenever it starts
sync in two-phase mode, but with only directional mappings no folder
runs the two-phase protocol, so the marker was never written and up
hung until its handoff timeout (caught by the kind e2e's reconcile with
up+down mappings). The marker signals "bootstrap phase concluded", not
"two-phase ran" — write it whenever the start mode was two-phase.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… roots

Requiring users to hand-write spec.volumes + podTemplate volumeMounts
for every extra sync mapping was needless friction. okdev now
provisions an emptyDir automatically at any additional remote root not
already covered by a target-container volume mount (a user PVC mount at
or above the root is respected and used instead), and the sidecar
mirror makes it servable by the remote syncthing. Users configure
mappings with paths only.

The additional remote roots participate in the workload snapshot
(omitempty — single-mapping sessions keep the legacy hash), so adding
or removing a mapping surfaces as drift and the familiar `up
--reconcile` flow recreates the pod with the new volume. The shared-
volume startup probe remains as the safety net.

Kind e2e drops the manual volume YAML, asserts plain `up` surfaces
reconcile guidance after adding a mapping, and reconciles into a
working two-direction setup.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@acmore acmore merged commit 36985e5 into main Jul 5, 2026
2 checks passed
@acmore acmore deleted the feat/sync-multi-folder branch July 5, 2026 12:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant